Dual-arm humanoid robot based on physical artificial intelligence and method for controlling the same

Abstract

The present disclosure relates to a physical artificial intelligence-based dual-arm humanoid robot and a method for controlling the same. The physical artificial intelligence-based dual-arm humanoid robot according to the present disclosure comprises a mobile base, a dual-arm manipulator installed on the mobile base and having 7 degrees of freedom and equipped with a force / torque sensor, a modular end effector mounted on the end of the dual-arm manipulator, a memory storing at least one process for performing an operation to adaptively control the grip strength and operation speed of the modular end effector, and a processor that performs an operation to adaptively control the grip strength and operation speed of the modular end effector based on the at least one process. The processor measures data through the force / torque sensor when an object is grasped using the end effector, estimates the properties of the object in real time based on the measured data, and dynamically adjusts an impedance control parameter based on the estimated properties of the object to adaptively control at least one of the grip strength and operation speed of the end effector.

Description

Technology Field

[0001] The present disclosure relates to a physical artificial intelligence-based dual-arm humanoid robot and a method for controlling the same. More specifically, the present disclosure relates to a physical artificial intelligence-based dual-arm humanoid robot capable of manipulating various types of end effectors through a physical artificial intelligence (Physical-AI)-based single-modality autonomous operation model and a method for controlling the same. Background Technology

[0002] Conventional robot systems have evolved to support tasks in various fields such as industry, logistics, and agriculture, but they still have limitations in terms of flexibility, adaptability, and autonomy. Since existing robots rely heavily on complex manual coding, specifically heuristic coding, the cost of customizing them for new applications is high and the rapid adoption of systems is difficult.

[0003] In particular, conventional systems have a fundamental limitation in that separate control models are required for each different end-effector. For example, since a 2-finger parallel gripper, a 3-finger robot hand, and a 5-finger humanoid robot hand each have different degrees of freedom (DOF), kinematic structures, and control parameters, a separate inverse kinematics solver, control algorithm, and training dataset are required for each type of end-effector, which can lead to exponentially increased development costs.

[0004] In addition, system expansion is very difficult because the entire control system must be redesigned and retrained whenever a new end effector (e.g., four-fingered robot hand, special purpose gripper) is added.

[0005] In addition, since gripping demonstration data collected with a 2-finger gripper cannot be used with a 3-finger or 5-finger robot hand, data must be collected and trained from scratch every time the end effector is changed.

[0006] In addition, for robot systems that support multiple end effectors, multiple control models corresponding to each end effector must be maintained in memory and switched in real time, which consumes a large amount of computational resources and causes delays during switching.

[0007] In addition, since independent models must be managed for each end effector, there is a burden of having to manage all models individually when updating models, fixing bugs, or improving performance.

[0008] In terms of mobility, fixed manipulators have a limited working range, and bipedal humanoid robots face limitations in practical application due to insufficient stability and economic feasibility.

[0009] Various industrial sectors, including logistics, manufacturing, and agriculture, remain highly dependent on manual labor, and the issue of labor shortages is becoming increasingly severe. However, existing robots struggle to effectively respond to irregular and dynamic work environments and fail to perceive and adapt to the physical characteristics of work objects (stiffness, weight, surface properties, etc.) in real time. Consequently, problems such as damage when handling soft objects or slippage when gripping rigid objects occur frequently.

[0010] Furthermore, in tasks requiring the collaboration of multiple robots, such as the transfer of large objects or complex assembly, efficient execution with a single robot is difficult. Additionally, the performance of existing robots degrades rapidly due to environmental factors like lighting, temperature, and changes in the workpiece, and retraining to compensate for this requires significant time and cost. Along with this, the inability to detect mechanical wear or component degradation in advance leads to unexpected breakdowns, frequently resulting in production stoppages.

[0011] Therefore, there is a need for a physical AI-based humanoid robot system capable of integrated control through a single AI model regardless of the type of end effector, autonomously adapting to environmental changes and the physical characteristics of objects, and enabling precise manipulation, mobility, and collaboration through dual-arm movements. Prior art literature

[0012] Registered Patent Publication No. 10-2385127 (Published April 6, 2022) The problem to be solved

[0013] One objective of the present disclosure is to provide a physical artificial intelligence-based dual-arm humanoid robot and a method for controlling the same, which reduces system complexity and increases scalability by integrating and controlling various types of end effectors (two-finger grippers, three-finger / four-finger / five-finger robot hands, etc.) through a single modality AI model, autonomously adapts to environmental changes and the physical characteristics of objects, and enables precise manipulation, mobility, and collaboration through dual-arm movements.

[0014] The problems that this disclosure aims to solve are not limited to those mentioned above, and other unmentioned problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0015] A physical artificial intelligence-based dual-arm humanoid robot according to the present disclosure for achieving the technical problem described above comprises a mobile base, a dual-arm manipulator installed on the mobile base and having 7 degrees of freedom and equipped with a force / torque sensor, a modular end effector mounted on the end of the dual-arm manipulator, a memory storing at least one process for performing an operation to adaptively control the grip strength and operation speed of the modular end effector, and a processor that performs an operation to adaptively control the grip strength and operation speed of the modular end effector based on the at least one process, wherein the processor measures data through the force / torque sensor when an object is grasped using the end effector, estimates the properties of the object in real time based on the measured data, and dynamically adjusts an impedance control parameter based on the estimated properties of the object to adaptively control at least one of the grip strength and operation speed of the end effector.

[0016] In addition, the control method of a physical artificial intelligence-based dual-arm humanoid robot according to the present disclosure may include the step of measuring data through a force / torque sensor when grasping an object using an end effector, the step of estimating the properties of the object in real time based on the measured data, and the step of adaptively controlling at least one of the grip strength and operation speed of the end effector by dynamically adjusting an impedance control parameter based on the estimated properties of the object.

[0017] In addition to this, a computer program stored on a computer-readable recording medium for implementing the present disclosure may be further provided.

[0018] In addition to this, a computer-readable recording medium for recording a computer program for implementing the present disclosure may be further provided. Effects of the invention

[0019] According to the aforementioned means for solving the problem of the present disclosure, the present disclosure can reduce system complexity and reduce customization costs by integrally controlling various end effectors through a single modality AI model.

[0020] In addition, the present disclosure can automate complex and variable manual tasks through Physical-AI-based autonomous operation.

[0021] In addition, the present disclosure can perform dual-arm manipulation operations while ensuring stability and efficiency through an omnidirectional rotary drive wheeled mobile platform.

[0022] In addition, the present disclosure can realize safe and precise operation by estimating the physical characteristics of an object in real time through adaptive impedance control and dynamically adjusting control parameters.

[0023] In addition, the present disclosure enables multiple robots to efficiently perform collaborative tasks while sharing a workspace through a multi-agent collaboration framework.

[0024] In addition, the present disclosure can automatically determine the optimal work sequence considering environmental information and maximize energy efficiency through context-aware work optimization.

[0025] In addition, the present disclosure can adapt to environmental changes in real time and maintain performance without retraining through online domain adaptation.

[0026] In addition, the present disclosure can minimize production downtime by predicting failures in advance through self-diagnosis and predictive maintenance.

[0027] In addition, the present disclosure can provide a robot system with a modular structure that can be rapidly applied to various sites.

[0028] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing

[0029] FIG. 1 is a perspective view showing a physical artificial intelligence-based dual-arm humanoid robot according to the present disclosure. FIG. 2 is a side view showing a physical artificial intelligence-based dual-arm humanoid robot according to the present disclosure. FIG. 3 is a block diagram for explaining a physical artificial intelligence-based dual-arm humanoid robot according to the present disclosure. FIG. 4 is a flowchart illustrating a representative control method of a physical artificial intelligence-based dual-arm humanoid robot according to the present disclosure. FIGS.S. 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, and 16 are flowcharts for explaining the control method examined in FIG. 4 in more detail. Specific details for implementing the invention

[0030] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and general content in the art to which this disclosure pertains or content that overlaps between embodiments is omitted. The terms 'part, module, component, block' as used in the specification may be implemented in software or hardware, and depending on the embodiments, a plurality of 'parts, modules, components, blocks' may be implemented as a single component, or a single 'part, module, component, block' may include a plurality of components.

[0031] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.

[0032] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0033] Throughout the specification, when it is stated that a component is located "on" another component, this includes not only cases where a component is in contact with another component, but also cases where another component exists between the two components.

[0034] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0035] Singular expressions include plural expressions unless there is an obvious exception in the context.

[0036] In each step, identification codes are used for convenience of explanation and do not describe the order of the steps; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.

[0037] The operating principles and embodiments of the present disclosure will be described below with reference to the attached drawings.

[0038] FIG. 1 is a perspective view showing a physical artificial intelligence-based dual-arm humanoid robot according to the present disclosure, FIG. 2 is a side view showing a physical artificial intelligence-based dual-arm humanoid robot according to the present disclosure, and FIG. 3 is a block diagram for explaining a physical artificial intelligence-based dual-arm humanoid robot according to the present disclosure.

[0039] The robot (100) according to the present disclosure relates to the field of robots based on artificial intelligence (AI) technology, and more specifically, it can operate various types of end effectors through a Physical-AI-based single modality autonomous operation model, safely operate target objects of various physical properties through adaptive impedance control, and multiple robots can perform cooperative work through a multi-agent collaboration framework.

[0040] To this end, with reference to FIG. 1 and FIG. 3, a physical artificial intelligence-based dual-arm humanoid robot (100) according to the present disclosure (hereinafter referred to as "robot (100) according to the present disclosure") may include a mobile base (110), a dual-arm manipulator (or manipulator) (120) installed on the mobile base and having 7 degrees of freedom and equipped with a force / torque sensor, a modular end effector (or end effector) (130) mounted on the end of the dual-arm manipulator, a memory storing at least one process for performing an operation to adaptively control the grip strength and operation speed of the modular end effector, and a processor (180) that performs an operation to adaptively control the grip strength and operation speed of the modular end effector based on the at least one process.

[0041] The memory can store data supporting various functions of the device and programs for the operation of the processor, and can store input / output data (e.g., music files, still images, videos, etc.), and can store multiple application programs (or applications) running on the device, data for the operation of the device, and instructions. At least some of these application programs may be downloaded from an external server via wireless communication.

[0042] Additionally, the memory may store at least one process (or task, operation, function, control method, process, data, algorithm, program, etc.) or processor that performs the method according to the present disclosure. Such at least one process may be performed under the control of the processor (180) and may refer to information used by the processor (180) to implement the method according to the present disclosure.

[0043] Such memory may include at least one type of storage medium among flash memory type, hard disk type, SSD type (Solid State Disk type), SSD type (Silicon Disk Drive type), multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (random access memory; RAM), SRAM (static random access memory), ROM (read-only memory; ROM), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk. Additionally, the memory may be a database that is separate from the device but connected via wired or wireless connection.

[0044] The processor (180) may be implemented as a memory that stores data for an algorithm or a program that reproduces the algorithm for controlling the operation of components within the device, and at least one processor that performs the aforementioned operation using the data stored in the memory. In this case, the memory and the processor may each be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.

[0045] In addition, the processor may control one or a combination of the components described above in order to implement various embodiments according to the present disclosure described below on the device.

[0046] All configurations described in this disclosure (e.g., ~~module, ~~means, ~~part, ~~system, etc.) may be implemented in hardware. In addition, these configurations may be implemented in separate physical hardware as well as in software. When implemented in software, they may be componentized and implemented in a block form within the process (180).

[0047] Referring to FIGS. 1 to 3, the mobile base (110) may include a wheeled drive unit (112) and a power and thermal management system (114).

[0048] The wheel-type drive unit (112) may have an omnidirectional movement mechanism applied to enable omnidirectional movement using a Mecanum wheel or an omniwheel.

[0049] The wheel-type drive unit (112) can independently perform forward and backward movement, left and right movement, diagonal movement, and rotation in place, and may have a high-load frame structure composed of a high-strength aluminum alloy or carbon fiber composite frame that supports the two-arm manipulator (120) and the upper body structure.

[0050] The wheel-type drive unit (112) may include an active damping system (or active suspension device) to absorb vibrations occurring during operation and maintain the precision of the manipulator.

[0051] The power and thermal management system (114) is composed of a lithium-ion or lithium-polymer battery and may include a high-capacity battery pack capable of continuous operation for at least 4 hours.

[0052] Additionally, the power and thermal management system (114) may include an energy regeneration system that recovers kinetic energy generated during manipulator descent and deceleration into electrical energy to charge the battery.

[0053] Additionally, the power and thermal management system (114) may include an intelligent thermal management system for maintaining an optimal operating temperature by monitoring the temperature of the drive unit and control computer in real time and dynamically adjusting the cooling fan speed.

[0054] The two-arm manipulator (120) refers to the mechanical arm part of a robot that physically manipulates a work object and can have a multi-joint structure like a human arm.

[0055] The dual-arm manipulator (120) may include links, joints, end-effectors, actuators, sensors, etc. Links are mechanical parts that connect the joints, and joints can adjust their position through rotation or linear movement. End-effectors are tools that perform specific tasks, such as grippers or welders, and actuators can drive the manipulator through motors or cylinders.

[0056] The dual-arm manipulator (120) may have a 7-axis degree-of-freedom structure.

[0057] For example, the two-arm manipulator (120) can be implemented with a biomimetic joint arrangement to have a range of motion similar to a human arm, with a configuration of three axes (pitch, roll, yaw) for the shoulder, one axis (pitch) for the elbow, and three axes (pitch, roll, yaw) for the wrist.

[0058] The dual-arm manipulator (120) is equipped with a servo motor combined with a high-precision harmonic drive or a cycloid reduction gear at each joint, enabling high-precision servo driving.

[0059] The dual-arm manipulator (120) has a built-in inverse kinematics solver that calculates each joint angle for the target position and posture in real time and can perform the role of an inverse kinematics analyzer.

[0060] The two-arm manipulator (120) may include a vertical lift system (122).

[0061] The vertical lift system (122) may include a linear guide actuator that extends the working height range using a ball screw or timing belt-driven vertical moving device.

[0062] The vertical lift system (122) can optimize the working height by automatically adjusting the manipulator reference height according to the position of the work target, thereby minimizing joint load and maximizing the working space.

[0063] The two-arm manipulator (120) may include a force / torque sensor (124).

[0064] The two-arm manipulator (120) can measure external force and moment applied to the end effector in real time by using a 6-axis force / torque sensor (124) and mounting a 6-axis force / torque sensor on the wrist portion of each arm.

[0065] Additionally, the two-arm manipulator (120) can detect a collision by immediately detecting the occurrence of an unexpected external force through a force / torque sensor (124) to perform a collision avoidance action or an emergency stop.

[0066] The modular end effector (130) may include an interface module (132) that provides a standard interface.

[0067] The interface module (132) can be mechanically connected to enable quick replacement of the end effector with an ISO 9409-1 or proprietary standard flange interface, and can be electrically connected to enable plug-and-play with a multi-pin connector that transmits power, communication, and sensor signals.

[0068] Additionally, the interface module (132) can automatically recognize the type, model, and control parameters of the connected end effector and perform automatic recognition to reflect them in the control model.

[0069] The modular end effector (130) can support (connect) various end effectors.

[0070] The modular end effector (130) can be connected to a 2-finger parallel gripper, which is a parallel opening and closing gripper suitable for simple gripping tasks; a 3-finger robot hand capable of stably fixing an object through triangular gripping; a 4-finger robot hand capable of manipulating objects of complex shapes and precise gripping; a 5-finger humanoid robot hand capable of fine manipulation and tool use with a structure similar to a human hand; and end effectors in the form of work-specialized tools such as a vacuum suction pad, a magnetic gripper, and a screw tightening tool.

[0071] The processor (180) may include an adaptive impedance control module (182).

[0072] The adaptive impedance control module (or control system) (182) can estimate the physical properties of an object in real time.

[0073] To this end, the adaptive impedance control module (182) can perform initial contact probing to estimate surface stiffness, friction coefficient, and deformation characteristics while applying a minute pressure change when the end effector first contacts the object.

[0074] Additionally, the adaptive impedance control module (182) can perform multi-sensor fusion estimation to estimate the center of gravity, mass, and moment of inertia of an object by fusing force / torque sensor data, joint torque feedback, and tactile sensor information.

[0075] In addition, the adaptive impedance control module (182) can perform material classification by automatically determining the material of an object (metal, plastic, wood, fabric, etc.) through a machine learning-based classification model.

[0076] The adaptive impedance control module (182) can perform dynamic control parameter adjustment.

[0077] The adaptive impedance control module (182) can generate an impedance parameter map to determine the stiffness coefficient, damping coefficient, and inertia coefficient in real time through a lookup table or function approximation according to the estimated object properties.

[0078] The adaptive impedance control module (182) can automatically adjust the grip strength to apply minimum gripping force to soft objects to prevent damage, and apply sufficient gripping force to rigid objects to prevent slipping.

[0079] The adaptive impedance control module (182) can perform operation speed adaptation to limit acceleration when the object has high inertia and to conservatively set the movement speed and acceleration when the object is fragile.

[0080] The adaptive impedance control module (182) can perform slip detection and re-grasp.

[0081] Specifically, the adaptive impedance control module (182) analyzes the high-frequency components of the force / torque sensor and the tactile sensor to detect early slippage of the object, and when slippage is detected, it can maintain a stable grip by gradually increasing the grip strength before the object falls.

[0082] Additionally, the adaptive impedance control module (182) can re-grasp in case of failure: if the object is completely lost, it can predict the falling trajectory of the object and attempt to re-grasp it, or place the object in a safe position and restart.

[0083] The processor (180) may include a multi-agent collaboration framework (184).

[0084] The multi-agent collaboration framework (184) may be implemented in hardware or may be implemented in a block form by componentizing it in software.

[0085] The multi-agent collaboration framework (184) may include a distributed work coordination system.

[0086] Specifically, the multi-agent collaboration framework (184) can divide the workspace within the factory or warehouse into a grid or zone and assign a dedicated area to each robot, and maximize work efficiency by reallocating the dedicated area in real time according to work density and robot workload, and perform priority-based coordination that redistributes resources among robots and dynamically adjusts the work order for urgent or high-priority tasks.

[0087] The multi-agent collaboration framework (184) can perform collaboration work protocols.

[0088] Specifically, the multi-agent collaboration framework (184) can perform a role assignment mechanism that automatically determines the role (support, assembly, inspection, etc.) of each robot in tasks requiring multiple robots (moving large objects, two-hand assembly).

[0089] In addition, the multi-agent collaboration framework (184) can synchronize the movements of each robot in time during collaborative work through synchronization control to prevent shaking or falling of the object, and can perform a force distribution algorithm that evenly distributes the load borne by each robot when jointly lifting the object or adjusts it in proportion to the robot's capabilities.

[0090] The multi-agent collaboration framework (184) can establish collision avoidance and path planning.

[0091] Specifically, the multi-agent collaboration framework (184) can share environment information recognized by each robot on a central server or in a distributed manner to update the integrated environment map in real time, and avoid predictive collisions by pre-adjusting its own movement path by considering the planned path and expected position of other robots, and can resolve deadlocks through priority rules or negotiation protocols when a deadlock occurs in which multiple robots are waiting for each other.

[0092] The multi-agent collaboration framework (184) can perform collective learning acceleration by uploading success / failure cases of tasks performed by each robot to a central database in a shared experience database so that the entire group of robots can share them, periodically perform model updates when new task data is accumulated and distribute learning updates to all robots, and perform transfer learning by transferring knowledge to other robots when one robot learns a new task to improve the learning speed of the entire system.

[0093] The processor (180) may include a context awareness module (186).

[0094] The context awareness module (186) can analyze the environment layout for context awareness-based task optimization.

[0095] Specifically, the context recognition module (186) can acquire three-dimensional spatial information of the work environment using an RGB-D camera and LiDAR, identify the position, orientation, and interrelationships (loading order, accessibility) of work target objects, and track dynamic obstacles: predict the trajectory of moving obstacles (dynamic obstacles) such as workers and forklifts to establish a safe work plan.

[0096] Additionally, the context recognition module (186) can automatically determine the sequence of operations.

[0097] To this end, the context recognition module (186) performs work order optimization (work priority evaluation) by comprehensively considering the work deadline, accessibility of objects, travel distance, etc., and readjusts the work order in real time when unexpected situations (object missing, obstacle occurrence) occur, and can handle constraints by automatically considering order constraints such as the fact that a specific object can only be accessed after another object, and time constraints for temperature-sensitive objects.

[0098] The context recognition module (186) can optimize energy efficiency by simultaneously optimizing the movement path of the mobile base and the operation trajectory of the manipulator to minimize total energy consumption, utilize inertia to utilize the momentum of the manipulator to reduce the energy required for the initial acceleration of the next movement, and manage standby power by switching unnecessary sensors and actuators to a low-power mode when there is no work.

[0099] The context awareness module (186) can predict the remaining work time, such as determining whether completion is possible based on the current battery level and planned work history, for battery management and automatic charging, determine the optimal charging time by considering work priority and charging time (charging scheduling), and perform automatic docking by driving autonomously to a charging station when charging is required, docking at the correct location, and resuming work after charging is completed.

[0100] The processor (180) may include a physical AI autonomous operation system (188).

[0101] The physical AI autonomous operation system (188) can perform fused motion capture.

[0102] The physical AI autonomous operation system (188) includes a motion capture glove that measures the bending angle of each finger joint, wrist position, and grip strength in real time, and a 3D vision system that includes a stereo camera or RGB-D camera that recognizes the 3D shape, position, and position of the work environment and target object. It can fuse sensor data to generate accurate work demonstration data by synchronizing the glove sensor and camera image in time and aligning the spatial coordinate system, and can perform multi-view shooting by simultaneously shooting from multiple angles to solve the occlusion problem and improve the 3D reconstruction accuracy.

[0103] The physical AI autonomous operation system (188) may include a single modality AI model structure.

[0104] Here, a single modality AI model can be an artificial intelligence model (189).

[0105] A single modality AI model (189) of a physical AI autonomous operation system (188) has structural features for integrated control of various types of end effectors.

[0106] Specifically, a single modality AI model (189) generates control commands using the same input and output formats regardless of the type of end effector. To this end, the model includes the following structural features.

[0107] First, it includes an End-effector Type Embedding Layer. This layer converts the type of currently mounted end-effector (2-finger parallel gripper, 3-finger robot hand, 4-finger robot hand, 5-finger humanoid robot hand, vacuum suction pad, magnetic gripper, etc.) into a one-hot vector or a learnable embedding vector. The dimension of the embedding vector can be set within the range of 64 to 256 dimensions, and it abstracts and represents the physical characteristics of the end-effector (degrees of freedom, operating mode, sensor configuration, etc.).

[0108] Second, it includes a Unified State Encoder. This encoder receives camera images (RGB-D), robot joint states (angles, angular velocity), force / torque sensor data, and end-effector type embeddings as input and maps them into a common latent space. Vision information is extracted into feature vectors through a Convolutional Neural Network (CNN) or Vision Transformer, and robot state information is encoded through a Multi-Layer Perceptron (MLP) before being fused through an attention mechanism or concatenation.

[0109] Third, it includes a Conditional Policy Network. This network takes the output of the integrated encoder and the end-effector type embedding as conditional inputs and outputs the next action (joint angular velocity or end-effector velocity). The network can adopt a Transformer-based architecture or a Recurrent Neural Network (RNN) structure, enabling sequence prediction that considers time-series information.

[0110] Fourth, it includes an output mapping layer for each end effector. The policy network generates control commands in a unified output space (e.g., based on a 5-finger robot hand with maximum degrees of freedom), and the output mapping layer selectively extracts and scales commands according to the actual degrees of freedom of the currently mounted end effector. For example, for a 2-finger parallel gripper, only 1D open / close commands are extracted, while for a 5-finger robot hand, the entire multidimensional commands are utilized.

[0111] Fifth, it includes a Physical Constraint Layer. Physical constraints such as joint limits, velocity limits, and torque limits of each end effector are stored in a database, and clipping or normalization is performed so that the output control commands do not violate these constraints.

[0112] Through this single-modality structure, the system enables immediate control by simply updating end-effector type information whenever an end-effector is replaced, without the need to develop or retrain a new control model. This significantly improves the system's scalability and maintainability.

[0113] In addition, during the learning process of a single modality AI model (189), work data using various end effectors is mixed and learned, thereby sharing common operational knowledge among end effectors (e.g., approach strategy, gripping timing) and simultaneously learning each end effector-specific skill (e.g., precision gripping of the five-finger hand, control of suction force of the vacuum pad).

[0114] Training data is collected in a balanced manner for each end effector, and insufficient data is supplemented through data augmentation techniques. For example, if there is relatively more data for 2-finger grippers, the sampling weight for 5-finger robot hand data is increased, or additional data is generated through simulation.

[0115] In the inference phase, the currently mounted end effector is identified through an automatic end effector recognition function, and the embedding vector of the corresponding type is input into the model. End effector recognition can be performed via electrical signals (connector pin placement), RFID tags, or vision-based recognition.

[0116] The physical AI autonomous operation system (188) receives camera images and robot status (joint angles, end effector positions) as input, encodes them into a common latent space, and can add the type of currently mounted end effector to the model input by representing it as a one-hot vector or an embedding vector.

[0117] Additionally, the physical AI autonomous operation system (188) includes a deep neural network that outputs the next action (joint angular velocity or end effector velocity) from encoded state information, and

[0118] Joint commands for various end effectors from 2 to 5 fingers can be generated in a unified output space and mapped to the corresponding end effectors at the time of execution.

[0119] An artificial intelligence model (189) that is a single modality AI model can be trained according to the following three-step learning process.

[0120] The artificial intelligence model (189) can be learned by imitation in one step.

[0121] Specifically, the artificial intelligence model (189) can collect state-action pair data by having a skilled worker wear motion capture gloves and perform actual work multiple times, learn the collected demonstration data in a supervised learning manner to obtain a basic policy that mimics the worker's behavior, and perform data augmentation to increase the diversity of the training data and improve generalization performance by applying noise addition, time distortion, spatial transformation, etc. to the demonstration data.

[0122] The artificial intelligence model (189) can be reinforced in two stages.

[0123] Specifically, the artificial intelligence model (189) can build a simulation environment by accurately modeling a robot, an object, and an environment in a physics engine-based simulator, design a reward system that provides high rewards when work goals are achieved and penalties when objects are damaged or collided, improve policies by going through millions of trials and errors in the simulation through deep reinforcement learning algorithms (PPO, SAC, etc.), and learn by randomly changing object positions, attitudes, lighting conditions, etc. to obtain robust policies and learn various scenarios.

[0124] The artificial intelligence model (189) can perform domain adaptation in three stages.

[0125] To increase the efficiency of single modality learning, the artificial intelligence model (189) may adopt a multi-task learning framework.

[0126] Specifically, tasks using different end effectors are defined as separate tasks, but the lower layer (integrated state encoder) is shared among all tasks, while the upper layer (policy network and output layer) can be partially shared or separated by task.

[0127] Through this structure, common feature extraction capabilities are shared among all end effectors, improving data efficiency, while control strategies specialized for specific end effectors are learned individually, ensuring performance.

[0128] In addition, by utilizing transfer learning, previously learned end-effector control knowledge can be applied to a new end-effector. For example, when applying a model trained on a three-fingered robot hand to a four-fingered robot hand, the training time for the new end-effector can be reduced by using most parameters as initial values ​​and fine-tuning with only a small amount of additional data.

[0129] Furthermore, by introducing meta-learning techniques, the ability to rapidly adapt with only a small amount of data can be learned in advance when new end-effectors are added. To this end, Model-Agnostic Meta-Learning (MAML) or Reptile algorithms can be utilized, and generalization ability is enhanced by training on adaptation tasks for various end-effector combinations on an episode-by-episode basis.

[0130] The artificial intelligence model (189) can perform domain randomization to enhance adaptability to the real environment by quantifying the difference between the simulation and the actual robot's sensor characteristics and physical response, analyzing the difference between the simulation and the actual, and learning by randomly setting the simulation parameters (friction, mass, sensor noise) over a wide range.

[0131] Additionally, the artificial intelligence model (189) can be trained to adapt to the actual environment by performing small-scale additional learning on the actual robot to correct the residual difference between the simulation and the actual, and by gradually increasing the difficulty from simple tasks to complex tasks.

[0132] The robot (100) according to the present disclosure may include an online domain adaptation module (140).

[0133] The online domain adaptation module (140) may be included in the processor (180).

[0134] The online domain adaptation module (140) may include an online domain adaptation mechanism.

[0135] Specifically, the online domain adaptation module (140) can automatically verify work results and determine whether a goal has been achieved through a vision system to automatically collect failure cases, perform automatic work success determination, and perform failure type classification, which automatically classifies and labels failure causes such as slipping, collision, gripping failure, and trajectory deviation, and store context information that records sensor data, robot status, and environmental conditions at the time of failure together so that they can be reproduced and analyzed.

[0136] The online domain adaptation module (140) may include a meta-learning framework that pre-applies meta-learning techniques to enable rapid adaptation with a small amount of new data in order to perform online policy fine-tuning.

[0137] The online domain adaptation module (140) can fine-tune some parameters of the policy network online using failure case data, perform incremental parameter updates that apply conservative updates to maintain existing performance, and apply the updated policy to actual operations after verifying it in a simulation or safe area.

[0138] The online domain adaptation module (140) can detect environmental changes (distribution shift detection) by monitoring changes in the statistical distribution of sensor data for environmental changes such as lighting, object placement, and temperature, and adapt to environmental changes (adaptive normalization) by updating the normalization parameters (mean, variance) of the input data online, and can mitigate rapid performance changes and perform stable operations (temporal ensemble) by weighted averaging policies from multiple past points in time.

[0139] The robot (100) according to the present disclosure may include a self-diagnosis system (150).

[0140] A self-diagnostic system (150) (or a self-diagnostic and predictive maintenance system) may be included in the processor (180).

[0141] The self-diagnosis system (150) can perform multi-sensor integrated monitoring to measure and record the driving current, temperature, vibration, and position deviation of each joint in real time for condition monitoring, and perform normal operation profile learning to set a normal range baseline by learning sensor data patterns during normal operation, and can perform abnormal sign detection to determine an abnormal sign if real-time sensor data deviates from the normal range or an unusual pattern occurs.

[0142] The self-diagnosis system (150) can estimate the wear condition by tracking the total operating time, number of rotations, and load history of each joint and drive unit during the accumulated usage period, and by estimating the degree of wear of the bearing through vibration frequency analysis and predicting the remaining lifespan.

[0143] The self-diagnostic system (150) can detect (measure) an increase in gear backlash by monitoring the increase in delay and error between joint commands and actual positions as a gear backlash measurement, and can detect a decrease in belt tension through a decrease in position accuracy and an increase in vibration in the timing belt drive system.

[0144] The self-diagnosis system (150) performs a trend analysis to predict the rate of performance degradation by analyzing the trend of change in performance indicators (position accuracy, response speed, energy efficiency) over time, and predicts the probability and timing of failure of a specific part by learning past maintenance history and failure data using a machine learning-based prediction model, and can generate a warning if the predicted remaining life falls below a set threshold.

[0145] The self-diagnosis system (150) provides step-by-step notifications according to severity, such as caution (yellow), warning (orange), and urgent (red), for automatic notification and maintenance history management, automatically notifies the manager of the predicted replacement time and list of necessary parts, automatically records all maintenance work history, replacement parts, and work time to use for statistical analysis and improvement of prediction models, and allows the manufacturer or expert to remotely check the robot status and provide diagnostic advice.

[0146] The robot (100) according to the present disclosure may include a multi-sensor system (160).

[0147] The multi-sensor system (160) may include a vision sensor (162), a distance sensor (164), an inertial and position sensor (166), and a tactile and proximity sensor (168).

[0148] The vision sensor (!60) may include an RGB-D camera that provides color information and depth information simultaneously to recognize objects and estimate their 3D positions, a stereo camera that generates precise distance measurements and stereoscopic images using binocular parallax, and a wide-angle camera that monitors the entire surrounding environment with a wide field of view to detect workers and obstacles.

[0149] The distance sensor (164) may include a LiDAR that recognizes long-distance obstacles and environmental structures with a 360-degree rotating laser scanner, an ultrasonic sensor for detecting short-distance obstacles and preventing collisions, and a Time of Flight (ToF) sensor that detects dynamic obstacles with a fast response speed at short distances.

[0150] The inertial and position sensor (166) may include an IMU (Inertial Measurement Unit) that measures the robot's attitude, tilt, and acceleration using a 3-axis accelerometer and a 3-axis gyroscope, a wheel encoder that measures the amount of wheel rotation to measure the distance traveled and the speed, and a GPS (Global Positioning System) / RTK (Real-Time Kinematic) that measures the precise position during outdoor work (SLAM-based position estimation indoors).

[0151] The tactile and proximity sensor (168) may include a tactile sensor array that measures contact distribution and pressure by arranging pressure sensors on the surface of the end effector finger, and a proximity sensor that detects when the distance to an object becomes close and decelerates or stops just before a collision.

[0152] The robot (100) according to the present disclosure may include an integrated control system (170) (or an End-to-End integrated control system).

[0153] The integrated control system (170) may include a status monitoring module (172), a work / path planning module (174), and a safety control module (176).

[0154] The status monitoring module (172) includes a dashboard interface that visualizes the location, work progress, battery level, and sensor status of each robot in real time, automatically records completed tasks, time taken, success rate, etc., provides statistical analysis, and can immediately send a notification to the manager when robot failure, work failure, safety risk, etc. are detected.

[0155] The work / path planning module (174) can optimally distribute multiple work requests among robots to establish a work plan to minimize the total completion time, readjust the work plan in real time when urgent work is added or robots fail, and perform priority-based scheduling that considers work importance, deadlines, customer requirements, etc.

[0156] Additionally, the work / path planning module (174) can perform SLAM-based map generation to automatically generate and update a 2D or 3D map of the work environment using sensor data for path planning, and calculate the shortest distance or shortest time path from the starting point to the destination to establish a global path plan.

[0157] Additionally, the task / path planning module (174) can establish a local path plan to follow a global path while avoiding dynamic obstacles detected in real time for path planning, and can perform traffic management to apply traffic rules and resolve bottlenecks when multiple robots share a narrow passage.

[0158] The safety control module (176) can track the location of a person in the workspace in real time through a camera and LiDAR, set a minimum safety distance between the worker and the robot, and reduce speed or stop when the worker approaches.

[0159] Additionally, the safety control module (176) includes an emergency stop system that immediately stops all operations via a physical emergency stop button and a wireless emergency stop signal, and can ensure safety by immediately stopping and reversing in the event of an unexpected collision.

[0160] Additionally, the integrated control system (170) can perform remote control and intervention. Specifically, the integrated control system (170) supports remote monitoring in which an administrator checks real-time video and robot status from a remote location, and can provide a manual intervention mode in which an administrator manually controls the robot remotely to resolve the problem when a problem occurs during autonomous operation.

[0161] In addition, the integrated control system (170) may also provide a virtual reality interface that allows the user to observe the work environment from the robot's perspective through a VR headset and intuitively convey commands.

[0162] In summary, the processor (180) includes an adaptive impedance control module (182), and the adaptive impedance control module (182) may include: a contact initial probing means for estimating surface stiffness and friction coefficients by applying minute pressure changes when the end effector first contacts the object; a multi-sensor fusion estimation means for estimating the center of gravity, mass, and moment of inertia of the object by fusing data output from the force / torque sensor, joint torque feedback, and tactile sensor information; an impedance parameter determination means for determining the stiffness coefficient, damping coefficient, and inertia coefficient in real time according to the estimated object properties; and a slip detection and re-grip means for early detection of object slippage and stepwise increase of grip strength by analyzing the high-frequency component of the force / torque sensor.

[0163] A processor (180) includes a multi-agent collaboration framework (184), and the multi-agent collaboration framework (184) may include a workspace management means for dividing a workspace into a grid or zone and assigning a dedicated area to each robot and reallocating the area in real time according to work density, a collaborative work coordination means for automatically determining the role of each robot in a task requiring multiple robots, synchronizing the movements of each robot in time, and evenly distributing the load when jointly lifting an object, a collision avoidance and deadlock resolution means for pre-adjusting one's own movement path by considering the planned path of other robots and resolving a deadlock between multiple robots through priority rules, and a collective learning means for uploading success / failure cases of tasks performed by each robot to a shared database and performing model updates and distributing them to all robots when new task data is accumulated.

[0164] A processor (180) includes a context recognition module (186), and the context recognition module (186) may include an environment layout analysis means for acquiring three-dimensional spatial information of a work environment through a multi-sensor system and identifying the position, posture, and interrelationship of work target objects, a work sequence determination means for determining a work sequence based on work end time, accessibility of objects, movement distance, and sequence constraints, and readjusting the work sequence in real time when an event occurs, and an energy efficiency optimization means for simultaneously adjusting the movement path of a mobile base and the operation trajectory of a manipulator to minimize total energy consumption and determining an optimal charging time based on the current battery level and planned work history.

[0165] The processor (180) includes a physical AI autonomous operation system (188), and the physical AI autonomous operation system (188) may include a fused manual motion capture means using a camera and a motion capture glove, an imitation learning means for learning demonstration data of a worker through imitation learning, a reinforcement learning means for performing reinforcement learning in a simulation environment, and a domain adaptation means for adapting a model learned in a simulation to a real environment.

[0166] The physical AI autonomous operation system (188) may include an artificial intelligence model (189) that encodes camera images and robot states into a common latent space, represents the type of currently mounted end effector as an embedding vector and adds it to the model input, and outputs joint commands for each type of end effector.

[0167] A robot (100) (or processor (180)) according to the present disclosure further comprises an online domain adaptation module (140), wherein the online domain adaptation module (140) can automatically collect and classify failure cases occurring during actual work performance, fine-tune policy network parameters online using a small amount of failure case data, detect environmental changes by monitoring changes in the statistical distribution of sensor data, and update input data normalization parameters online.

[0168] A robot (100) (or processor (180)) according to the present disclosure further comprises a self-diagnosis system (150), wherein the self-diagnosis system (150) may include: a state monitoring means that measures the driving current, temperature, vibration, and position deviation of each joint in real time, learns the sensor data pattern during normal operation to set a normal range, and determines an abnormal sign if the real-time sensor data deviates from the normal range; a wear state estimation means that estimates the degree of bearing wear through vibration frequency analysis and detects an increase in gear backlash by monitoring an increase in the error between joint commands and actual positions; a performance degradation prediction means that analyzes the trend of change in performance indicators over time and learns past maintenance history and failure data to predict the probability and timing of failure of specific parts; and an automatic notification and history management means that generates a step-by-step alarm when the predicted remaining life falls below a threshold, automatically notifies the time for part replacement and a list of necessary parts, and automatically records all maintenance work details.

[0169] A robot (100) (or processor (180)) according to the present disclosure further includes an integrated control system (170), and the integrated control system (170) may include a status monitoring module that visualizes the location, work progress, battery level, and sensor status of each robot in real time and automatically records the history of completed work; a work planning module that optimally distributes multiple work requests among robots and readjusts the work plan in real time when an urgent work is added or a robot fails; a path planning module that automatically generates a work environment map based on SLAM, plans global and local paths, and performs traffic management among multiple robots; a safety control module that tracks the location of a person in the workspace in real time, maintains a safe distance and provides an emergency stop function, and immediately retreats in the reverse direction in the event of a collision; and a remote control module that allows an administrator to check real-time video and robot status from a remote location and manually control the robot if necessary.

[0170] The above work planning module and the above path planning module can be implemented by integrating them into the previously described work / path planning module (174).

[0171] The functions / operations / control methods performed by the previously described adaptive impedance control module (182), multi-agent collaboration framework (184), context awareness module (186), physical AI autonomous operation system (188), artificial intelligence model (189), online domain adaptation module (140), self-diagnosis system (150), and integrated control system (170) can be applied by analogy in the same / similar way as those performed by the processor (180).

[0172] FIG. 4 is a flowchart illustrating a representative control method of a physical artificial intelligence-based dual-arm humanoid robot according to the present disclosure, and FIG. 5, FIG. 6, FIG. 7, FIG. 8, FIG. 9, FIG. 10, FIG. 11, FIG. 12, FIG. 13, FIG. 14, FIG. 15 and FIG. 16 are flowcharts for explaining the control method examined in FIG. 4 in more detail.

[0173] Referring to FIG. 4, the processor (180) measures data from a force / torque sensor when gripping an object using an end effector (S410), estimates the properties of the object in real time based on the measured data (S420), and dynamically adjusts an impedance control parameter based on the estimated properties of the object to adaptively control at least one of the grip strength and operation speed of the end effector (S430).

[0174] Referring to FIG. 5, the processor (180) of the robot (100) according to the present disclosure may perform (or include) the following steps: (a) collecting manual demonstration data of a worker using motion capture gloves and a camera; (b) learning a basic work policy by performing imitation learning based on the collected demonstration data; (c) improving work success rate and efficiency through reinforcement learning in a simulation environment; (d) optimizing performance in a real environment by applying Sim2Real technology; (e) controlling various end effectors in an integrated manner using a single learned model; (f) estimating the stiffness, weight, and surface friction coefficient of an object in real time based on data measured from a 6-axis force / torque sensor; and (g) adaptively controlling grip strength and operation speed by dynamically adjusting impedance control parameters according to the estimated physical properties of the object.

[0175] With reference to FIG. 6, step (b) is described in detail. A processor (180) can perform the following steps: (b-1) extracting state-action pairs from demonstration data; (b-2) a data augmentation step that increases the diversity of the training data by applying noise addition, time distortion, and spatial transformation to the demonstration data; (b-3) a step of performing behavior replication learning with the extracted and augmented data; and (b-4) a step of evaluating and supplementing the performance of the learned policy.

[0176] With reference to FIG. 7, step (c) can be described in detail as follows: the processor (180) can perform (c-1) a step of modeling a robot, an object, and an environment in a physics engine-based simulator; (c-2) a step of designing a reward system that provides a high reward when a task goal is achieved and a penalty when an object is damaged or collides; (c-3) a step of optimizing a policy in a simulation through a deep reinforcement learning algorithm; and (c-4) a step of obtaining a robust policy by learning while randomly changing the object position, attitude, lighting conditions, etc.

[0177] With reference to FIG. 8, step (d) is described in detail. The processor (180) can perform (d-1) a step of quantifying the difference between the sensor characteristics and physical response of the simulation and the actual robot, (d-2) a domain randomization step of enhancing adaptability to the actual environment by learning by setting simulation parameters randomly over a wide range, (d-3) a step of correcting the residual difference between the simulation and the actual by performing small-scale additional learning on the actual robot, and (d-4) a step of adapting to the actual environment by gradually increasing the difficulty level from simple tasks to complex tasks.

[0178] With reference to FIG. 9, step (e) is described in detail. The processor (180) can perform the following steps: (e-1) receiving camera images and robot states as input and encoding them into a common latent space; (e-2) representing the types of currently mounted end effectors as embedding vectors and adding them to model inputs; (e-3) executing a policy network that outputs the next action from the encoded state information; and (e-4) generating joint commands for various end effectors from 2 to 5 fingers in a unified output space and mapping them to the corresponding end effectors at the time of execution.

[0179] With reference to FIG. 10, step (f) is described in detail. The processor (180) can perform the following steps: (f-1) estimating surface stiffness and friction coefficients while applying minute pressure changes when the end effector first contacts the object; (f-2) estimating the center of gravity, mass, and moment of inertia of the object by fusing force / torque sensor data, joint torque feedback, and tactile sensor information; and (f-3) automatically determining the material of the object through a machine learning-based classification model.

[0180] With reference to FIG. 11, step (g) is described in detail. The processor (180) can perform the following steps: (g-1) determining the stiffness coefficient, damping coefficient, and inertia coefficient in real time through a lookup table or function approximation according to the estimated physical properties of the object; (g-2) applying a minimum gripping force to a soft object and applying a sufficient gripping force to prevent slipping to a rigid object; (g-3) adaptively adjusting the acceleration and movement speed according to the inertia of the object; and (g-4) analyzing the high-frequency components of the force / torque sensor to detect the slipping of the object early and increasing the grip strength step by step when slipping is detected.

[0181] Additionally, referring to FIG. 12, the processor (180) of the robot (100) of the present disclosure may perform (h) a step of performing a collaborative task between a plurality of robots, (i) a task optimization step based on context awareness, (j) an online domain adaptation step, and (k) a self-diagnosis and predictive maintenance step.

[0182] With reference to FIG. 13, step (h) is described in detail. The processor (180) can perform the following steps: (h-1) dividing the workspace into a grid or zone and assigning a dedicated area to each robot, and rearranging the area in real time according to work density; (h-2) automatically determining the role of each robot in a task requiring multiple robots and synchronizing the movements of each robot in time; (h-3) evenly distributing the load borne by each robot when jointly lifting an object or adjusting it in proportion to the robot's capabilities; (h-4) pre-adjusting its own movement path by considering the planned path of other robots; and (h-5) uploading success / failure cases of tasks performed by each robot to a shared database and improving the performance of the entire system through collective learning.

[0183] With reference to FIG. 14, step (i) is described in detail, and the processor (180) can perform the following steps: (i-1) acquiring three-dimensional spatial information of the work environment using an RGB-D camera and LiDAR and identifying the position, orientation, and interrelationship of work target objects; (i-2) optimizing the work order by comprehensively considering the work completion time, accessibility of objects, travel distance, and order constraints; (i-3) readjusting the work order in real time when an unexpected situation occurs; (i-4) minimizing total energy consumption by simultaneously optimizing the movement path of the mobile base and the operation trajectory of the manipulator; and (i-5) determining whether completion is possible based on the current battery level and planned work history and determining the optimal charging time.

[0184] With reference to FIG. 15, step (j) is described in detail. The processor (180) can perform the following steps: (j-1) automatically verifying work results through a vision system and determining whether the goal has been achieved; (j-2) automatically classifying the cause of failure when work fails and recording sensor data, robot status, and environmental conditions at the time of failure; (j-3) using failure case data to fine-tune some parameters of the policy network online, while applying conservative updates to maintain existing performance; (j-4) monitoring changes in the statistical distribution of sensor data to detect environmental changes such as lighting, object placement, and temperature; and (j-5) updating the normalization parameters of the input data online to adapt to environmental changes.

[0185] With reference to FIG. 16, step (k) is described in detail. A processor (180) can perform the following steps: (k-1) measuring and recording the driving current, temperature, vibration, and position deviation of each joint in real time; (k-2) learning the sensor data pattern during normal operation to set a normal range baseline; (k-3) determining that if real-time sensor data deviates from the normal range or a specific pattern occurs, it is an abnormal sign; (k-4) estimating the degree of bearing wear through vibration frequency analysis and detecting an increase in gear backlash by monitoring the increase in error between the joint command and the actual position; (k-5) analyzing the trend of performance indicator change over time and learning past maintenance history and failure data to predict the probability and timing of failure of a specific part; (k-6) generating a step-by-step alarm when the predicted remaining life falls below a threshold value and automatically notifying the time for part replacement and a list of necessary parts.

[0186] According to the aforementioned means for solving the problem of the present disclosure, the present disclosure can reduce system complexity and reduce customization costs by integrally controlling various end effectors through a single modality AI model.

[0187] In addition, the present disclosure can automate complex and variable manual tasks through Physical-AI-based autonomous operation.

[0188] In addition, the present disclosure can perform dual-arm manipulation operations while ensuring stability and efficiency through an omnidirectional rotary drive wheeled mobile platform.

[0189] In addition, the present disclosure can realize safe and precise operation by estimating the physical characteristics of an object in real time through adaptive impedance control and dynamically adjusting control parameters.

[0190] In addition, the present disclosure enables multiple robots to efficiently perform collaborative tasks while sharing a workspace through a multi-agent collaboration framework.

[0191] In addition, the present disclosure can automatically determine the optimal work sequence considering environmental information and maximize energy efficiency through context-aware work optimization.

[0192] In addition, the present disclosure can adapt to environmental changes in real time and maintain performance without retraining through online domain adaptation.

[0193] In addition, the present disclosure can minimize production downtime by predicting failures in advance through self-diagnosis and predictive maintenance.

[0194] In addition, the present disclosure can provide a robot system with a modular structure that can be rapidly applied to various sites.

[0195] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below.

[0196] Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0197] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform a desired characteristic (or objective) are created by a basic artificial intelligence model being trained using a number of training data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0198] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained by the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0199] According to an exemplary embodiment of the present disclosure, a processor can implement artificial intelligence. Artificial intelligence refers to a machine learning method based on an artificial neural network that enables a machine to learn by mimicking human biological neurons. Methodologies of artificial intelligence can be classified according to the learning method into supervised learning, where input and output data are provided together as training data and the solution (output data) to the problem (input data) is predetermined; unsupervised learning, where only input data is provided without output data and the solution (output data) to the problem (input data) is not predetermined; and reinforcement learning, where a reward is given from an external environment whenever an action is taken from the current state, and learning proceeds in a direction that maximizes such reward. In addition, artificial intelligence methodologies can be classified according to the architecture, which is the structure of the learning model. The architectures of widely used deep learning technologies can be classified into Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Transformers, and Generative Adversarial Networks (GAN).

[0200] The device and system may include an artificial intelligence model. The artificial intelligence model may be a single model or may be implemented as multiple models. The artificial intelligence model may be composed of a neural network (or artificial neural network) and may include statistical learning algorithms in machine learning and cognitive science that mimic biological neurons. A neural network may refer to a model that possesses problem-solving capabilities by having artificial neurons (nodes) that form a network through synaptic connections and change the strength of synaptic connections through learning. The neurons of a neural network may include combinations of weights or biases. A neural network may include one or more layers composed of one or more neurons or nodes. For example, the device may include an input layer, a hidden layer, and an output layer. The neural network constituting the device can infer a result (output) to be predicted from an arbitrary input by changing the weights of the neurons through learning.

[0201] The processor can create neural networks, train or learn neural networks, perform computations based on received input data, generate information signals based on the results of the computation, or retrain neural networks. Neural network models may include, but are not limited to, various types of models such as Convolutional Neural Networks (CNN), Region with Convolutional Neural Networks (R-CNN), Region Proposal Networks (RPN), Recurrent Neural Networks (RNN), Stacking-based Deep Neural Networks (S-DNN), State-Space Dynamic Neural Networks (S-SDNN), Deconvolution Networks, Deep Belief Networks (DBN), Restricted Boltzmann Machines (RBM), Fully Convolutional Networks, Long Short-Term Memory Networks (LSTM), and Classification Networks, such as GoogleNet, AlexNet, and VGG Network. The processor may include one or more processors to perform computations according to neural network models. For example, a neural network is a deep neural network It may include a (Deep Neural Network).

[0202] Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto) Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), GAN (Generative Adversarial Network), LSM (Liquid State Machine), ELM (Extreme Learning Machine), ESN (Echo It will be understood by a person skilled in the art that any neural network may be included, but is not limited to, State Network, Deep Residual Network, Differential Neural Computer, Neural Turning Machine, Capsule Network, Kohonen Network, and Attention Network.

[0203] According to exemplary embodiments of the present disclosure, the processor comprises a Convolutional Neural Network (CNN) such as GoogleNet, AlexNet, VGG Network, Region with Convolutional Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based Deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restricted Boltzmann Machine (RBM), Fully Convolutional Network, Long Short-Term Memory Network (LSTM), Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC / QA, Text Analysis, Dialog System, GPT-3, GPT-4 for Natural Language Processing, Visual Analytics, Visual Understanding, Video Synthesis for Vision Processing, Anomaly Detection, Prediction, Time-Series Forecasting, Optimization, Recommendation for ResNet Data Intelligence, Various artificial intelligence structures and algorithms, such as data creation, may be used, but are not limited thereto.

[0204] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. The instructions may be stored in the form of program code and, when executed by a processor, may generate a program module to perform the operation of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0205] Computer-readable recording media include all types of recording media that store instructions that can be decoded by a computer. Examples include ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disk, flash memory, optical data storage devices, etc.

[0206] As described above, the disclosed embodiments have been explained with reference to the attached drawings. Those skilled in the art will understand that the present disclosure may be practiced in forms different from the disclosed embodiments without changing the technical spirit or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be interpreted restrictively.

Claims

Claim 1 A physical artificial intelligence-based dual-arm humanoid robot comprising: a mobile base; a dual-arm manipulator installed on the mobile base, having 7 degrees of freedom and equipped with a force / torque sensor; a modular end effector mounted on the end of the dual-arm manipulator; a memory storing at least one process for performing an operation to adaptively control the grip strength and operation speed of the modular end effector; and a processor that performs an operation to adaptively control the grip strength and operation speed of the modular end effector based on the at least one process, wherein the processor measures data through the force / torque sensor when an object is grasped using the end effector, estimates the properties of the object in real time based on the measured data, and dynamically adjusts an impedance control parameter based on the estimated properties of the object to adaptively control at least one of the grip strength and operation speed of the end effector. Claim 2 A physical artificial intelligence-based dual-arm humanoid robot according to claim 1, wherein the processor comprises an adaptive impedance control module, and the adaptive impedance control module comprises: a contact initial probing means for estimating surface stiffness and a friction coefficient while applying minute pressure changes when an end effector first contacts an object; a multi-sensor fusion estimation means for estimating the center of gravity, mass, and moment of inertia of an object by fusing data output from the force / torque sensor, joint torque feedback, and tactile sensor information; an impedance parameter determination means for determining a stiffness coefficient, a damping coefficient, and an inertia coefficient in real time according to the estimated physical properties of the object; and a slip detection and re-grasp means for analyzing high-frequency components of the force / torque sensor to detect object slippage early and gradually increase grip strength. Claim 3 A physical artificial intelligence-based dual-arm humanoid robot according to claim 1, wherein the processor comprises a multi-agent collaboration framework, and the multi-agent collaboration framework comprises: a workspace management means for dividing a workspace into a grid or zone, assigning a dedicated area to each robot, and reallocating the area in real time according to work density; a collaborative work coordination means for automatically determining the role of each robot in a task requiring multiple robots, synchronizing the movements of each robot in time, and evenly distributing the load when jointly lifting an object; a collision avoidance and deadlock resolution means for pre-adjusting one's own movement path by considering the planned path of other robots and resolving a deadlock between multiple robots through priority rules; and a collective learning means for uploading success / failure cases of tasks performed by each robot to a shared database, and performing model updates and distributing them to all robots when new task data is accumulated. Claim 4 A physical artificial intelligence-based dual-arm humanoid robot according to claim 1, wherein the processor comprises a context recognition module, and the context recognition module comprises: an environment layout analysis means for acquiring three-dimensional spatial information of a work environment through a multi-sensor system and identifying the position, posture, and interrelationship of work target objects; a work sequence determination means for determining a work sequence based on work completion time, object accessibility, movement distance, and sequence constraints, and readjusting the work sequence in real time upon the occurrence of an event; and an energy efficiency optimization means for simultaneously adjusting the movement path of a mobile base and the operation trajectory of a manipulator to minimize total energy consumption, and determining an optimal charging time based on the current battery level and planned work history. Claim 5 In claim 1, the processor comprises a physical AI autonomous operation system, and the physical AI autonomous operation system comprises: a fused manual motion capture means using a camera and a motion capture glove; an imitation learning means for learning demonstration data of a worker through imitation learning; a reinforcement learning means for performing reinforcement learning in a simulation environment; and a domain adaptation means for adapting a model learned in a simulation to a real environment, thereby forming a physical AI-based dual-arm humanoid robot. Claim 6 In claim 5, the physical AI autonomous operation system comprises a physical AI-based dual-arm humanoid robot including an artificial intelligence model that encodes camera images and robot states into a common latent space, represents the type of currently mounted end effector as an embedding vector and adds it to the model input, generates joint commands in a unified output space for each type of end effector, and maps the generated joint commands to the actual degrees of freedom of the corresponding end effector at the time of execution. Claim 7 A physical AI-based dual-arm humanoid robot according to claim 1, further comprising an online domain adaptation module, wherein the online domain adaptation module automatically collects and classifies failure cases occurring during actual task execution, fine-tunes policy network parameters online using a small amount of failure case data, detects environmental changes by monitoring changes in the statistical distribution of sensor data, and updates input data normalization parameters online. Claim 8 A physical artificial intelligence-based dual-arm humanoid robot according to claim 1, further comprising a self-diagnosis system, wherein the self-diagnosis system includes: a state monitoring means that measures the driving current, temperature, vibration, and position deviation of each joint in real time, learns sensor data patterns during normal operation to set a normal range, and determines an abnormal sign if real-time sensor data deviates from the normal range; a wear state estimation means that estimates the degree of bearing wear through vibration frequency analysis and detects an increase in gear backlash by monitoring an increase in the error between joint commands and actual positions; a performance degradation prediction means that analyzes the trend of change in performance indicators over time and learns past maintenance history and failure data to predict the probability and timing of failure of specific parts; and an automatic notification and history management means that generates a step-by-step alarm when the predicted remaining life falls below a threshold, automatically notifies the time for part replacement and a list of necessary parts, and automatically records all maintenance work details. Claim 9 A physical artificial intelligence-based dual-arm humanoid robot according to claim 1, further comprising an integrated control system, wherein the integrated control system includes: a status monitoring module that visualizes the location, work progress, battery level, and sensor status of each robot in real time and automatically records the history of completed work; a work planning module that optimally distributes multiple work requests among robots and readjusts the work plan in real time in the event of an additional urgent task or robot failure; a path planning module that automatically generates a work environment map based on SLAM, plans global and local paths, and performs traffic management among multiple robots; a safety control module that tracks the location of a person within the workspace in real time, provides safety distance maintenance and emergency stop functions, and immediately retreats in the reverse direction in the event of a collision; and a remote control module that allows a manager to check real-time video and robot status from a remote location and manually control the robot if necessary. Claim 10 delete

Images