Robot control device and control method for the same

Abstract

To provide a robot control device and a robot control method for inferring the intention of a user from the action of the user.SOLUTION: A robot control device includes a memory storing an instruction word executable by a computer, and one or more processors for accessing to the memory and executing the instruction word. A robot control method controls a robot to apply event information on the action of a user identified by user action data recognized from a robot and context information on a time space where the action occurs, to a knowledge graph formed by a relation between an event instance related to the event information and a context instance related to the context information, to apply the event instance included in the knowledge graph to a rule generation model for generating information on the intention of the user and acquire user intention data related to the intention of the user, and to perform a target work related to an expected action which is an action subsequent to the action on the basis of the user intention data.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a robot control device and a control method thereof, and more particularly to a technique for inferring a user's intention from the user's actions. [Background technology]

[0002] With the development of artificial intelligence and robotics technology, demand and interest in service robots are increasing. In particular, technology is being developed that utilizes service robots to provide services to users (e.g., customers) in non-face-to-face environments. As a result, there is an increasing need for intelligent service robots that can perform tasks at an advanced level approaching that of humans. In particular, there is a demand for robots that can infer and provide the services required by customers based on their behavior and situation information within a specific space.

[0003] To solve these technical problems, it is necessary to develop technology that can gradually infer human behavioral information (e.g., walking) based on customer behavior and situational information, and ultimately control the robot by inferring the human's intentions. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-18893 Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention has been made in view of the above-mentioned conventional techniques, and an object of the present invention is to provide a robot control device and a control method thereof that infer a user's intention from the user's actions. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided a robot control device that includes a memory storing computer-executable instructions and at least one processor that accesses the memory and executes the instructions. The at least one processor applies event information related to a user's behavior identified in user behavior data perceived by the robot and context information related to the time and space in which the behavior occurred to a knowledge graph formed by a relationship between an event instance related to the event information and a context instance related to the context information, applies the event instance among the instances included in the knowledge graph to a rule generation model that generates information related to the intention of the behavior, thereby obtaining user intention data related to the intention of the behavior, and controls the robot to perform a target task related to a predicted behavior, which is a subsequent behavior of the behavior, based on the user intention data.

[0007] In one embodiment, the at least one processor identifies a predetermined Event-Condition-Action (ECA) rule from the rule generation model based on applying the event instance to the rule generation model, applies the event instance to the ECA rule to obtain a condition related to the event instance, and applies the context instance paired with the event instance to the condition to obtain the user intent data. In one embodiment, the at least one processor identifies, from the rule generation model, a Neural Compositional Rule Learning (NCRL)-based rule mining model that generates the user intention data based on the event instance among the instances included in the knowledge graph, regardless of the context instance, and updates the ECA rules based on the output of the rule mining model and the knowledge graph. In one embodiment, the at least one processor obtains behavior sequence data related to the event instance from the rule mining model, obtains at least one candidate context instance related to the behavior sequence data from the knowledge graph, and updates the ECA rule based on the behavior sequence data and the at least one candidate context instance. In one embodiment, the at least one processor identifies a first candidate context instance and a second candidate context instance different from the first candidate context instance from the at least one candidate context instance, determines a first count that is the number of objects that satisfy a combination between a location included in the first candidate context instance and the behavioral sequence data, determines a second count that is the number of objects that satisfy a combination between a location included in the second candidate context instance and the behavioral sequence data, and determines one of the first candidate context instance or the second candidate context instance as a target context instance for the behavioral sequence data based on a comparison between the first count and the second count. In one embodiment, the at least one processor updates the ECA rule by inserting a relationship between the behavioral sequence data and the target context instance into the ECA rule based on the determination of the target context instance related to the behavioral sequence data. In one embodiment, the at least one processor applies the instance of event information to the ECA rule based on the update of the ECA rule to obtain additional conditions related to the instance of event information, and applies an instance of context information paired with the event information to the additional conditions to obtain additional user intention data different from the user intention data. In one embodiment, the at least one processor determines a target task related to the predicted behavior based on the user intention data and the additional user intention data, and controls the robot to perform the target task. In one embodiment, the at least one processor generates Resource Description Framework (RDF) data, which is a data format compatible with the ontology of the knowledge graph and includes a relationship between the event information and the context information, and converts the RDF data into the event instances and the context instances and stores them in the knowledge graph.

[0008] To achieve the above object, according to one aspect of the present invention, a robot control method for a robot control device executed by at least one processor includes the steps of: applying event information related to a user's action identified in user action data received from a robot and context information related to the time and space in which the action occurred to a knowledge graph formed by a relationship between an event instance related to the event information and a context instance related to the context information; applying the event instance among the instances included in the knowledge graph to a rule generation model that generates information related to the intention of the action to obtain user intention data related to the intention of the action; and controlling the robot so that the robot performs a target task related to a predicted action, which is a subsequent action of the action, based on the user intention data.

[0009] In one embodiment, the robot control method further includes identifying a predetermined Event-Condition-Action (ECA) rule from the rule generation model based on applying the event instance to the rule generation model; applying the event instance to the ECA rule to obtain a condition related to the event instance; and applying the context instance paired with the event instance to the condition to obtain the user intention data. In one embodiment, the step of acquiring the user intention data includes identifying, from the rule generation model, a Neural Compositional Rule Learning (NCRL)-based rule mining model that generates the user intention data based on the event instances among the instances included in the knowledge graph, regardless of the context instances; and updating the ECA rules based on an output of the rule mining model and the knowledge graph. In one embodiment, acquiring the user intent data includes acquiring action sequence data related to the event instance from the rule mining model; acquiring at least one candidate context instance related to the action sequence data from the knowledge graph; and updating the ECA rules based on the action sequence data and the at least one candidate context instance. In one embodiment, the acquiring of the user intention data includes identifying a first candidate context instance and a second candidate context instance different from the first candidate context instance from the at least one candidate context instance; determining a first count, which is the number of objects that satisfy a combination between a location included in the first candidate context instance and the behavioral sequence data; determining a second count, which is the number of objects that satisfy a combination between a location included in the second candidate context instance and the behavioral sequence data; and determining one of the first candidate context instance or the second candidate context instance as a target context instance for the behavioral sequence data based on a comparison between the first count and the second count. In one embodiment, the step of acquiring the user intention data includes updating the ECA rule by inserting a relationship between the action sequence data and the target context instance into the ECA rule based on the determination of the target context instance related to the action sequence data. In one embodiment, the step of acquiring the user intention data includes: applying the instance of the event information to the ECA rule based on the update of the ECA rule to acquire additional conditions related to the instance of the event information; and applying an instance of context information paired with the event information to the additional conditions to acquire additional user intention data different from the user intention data. In one embodiment, the robot control method further includes determining a target task related to the predicted behavior based on the user intention data and the additional user intention data, and controlling the robot so that the robot performs the target task. In one embodiment, the robot control method further includes generating Resource Description Framework (RDF) data, which is a data format compatible with the ontology of the knowledge graph and includes a relationship between the event information and the context information, and converting the RDF data into the event instances and the context instances and storing them in the knowledge graph. [Effects of the Invention]

[0010] The effects of the robot control device and the control method thereof according to the present invention will be described below.

[0011] According to the robot control device and control method of the present invention, by inferring the user's intention through a rule generation model based on the event instances and context instances contained in the knowledge graph, it is possible to provide superior effects compared to inferring the intention from the user's actions alone by taking into account the user's actions and the circumstances in which the actions occurred.

[0012] In addition, according to the present invention, by updating ECA rules based on the output of the rule mining model and the knowledge graph, the strengths of knowledge-driven and data-driven access methods are combined to generate expressive rules, while at the same time reducing the time and cost required to acquire domain knowledge.

[0013] Furthermore, according to the present invention, rather than inferring a user's intention based on the output of a rule mining model, the user's intention is inferred based on user intention data obtained from ECA rules and additional user intention data obtained from updated ECA rules, thereby achieving the effect of increasing intention recognition performance by discovering rules that could not be discovered previously.

[0014] In addition, various other effects are provided that can be grasped directly or indirectly through this specification. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram illustrating a robot control device according to an embodiment of the present invention. [Figure 2] 1 is a flowchart illustrating a robot control method according to an embodiment of the present invention. [Figure 3] FIG. 2 is a block diagram showing a detailed configuration of the robot control device according to the embodiment of the present invention. [Figure 4] FIG. 10 is a diagram showing an example of user behavior data received from a robot in a robot control device according to an embodiment of the present invention. [Figure 5] FIG. 10 is a diagram showing an example of RDF data generated from user behavior data in a robot control device according to an embodiment of the present invention. [Figure 6] FIG. 10 is a diagram illustrating an example of an instance stored in a knowledge graph in a robot control device according to an embodiment of the present invention. [Figure 7] FIG. 10 is a diagram illustrating an example of acquiring user intention data through an ECA rule in a robot control device according to an embodiment of the present invention. [Figure 8] FIG. 10 is a diagram illustrating an example of an output of a rule mining model in a robot control device according to an embodiment of the present invention. [Figure 9] FIG. 10 is a diagram illustrating an example of an operation for updating an ECA rule in a robot control device according to an embodiment of the present invention. [Figure 10]FIG. 1 illustrates a computing system for a robot control device or method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, specific examples of embodiments of the present invention will be described in detail with reference to the drawings.

[0017] When assigning reference numerals to components in each drawing, care should be taken to use the same numerals to the same components even if they appear in different drawings. Furthermore, when describing embodiments of the present invention, if a detailed description of related known structures or functions is deemed to obscure understanding of the embodiments of the present invention, such detailed description will be omitted. In particular, various embodiments of the present invention will be described with reference to the drawings. However, this is not intended to limit the technology described herein to specific embodiments, and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present invention. In describing the drawings, similar reference numerals will be used to refer to similar components.

[0018] When describing components of embodiments of the present invention, terms such as "first," "second," "A," "B," "(a)," and "(b)" are used. These terms are intended to distinguish a component from other components and do not limit the nature, order, or sequence of the components. Furthermore, unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. Terms similar to those defined in commonly used dictionaries should be interpreted as meanings consistent with the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly defined herein. For example, terms such as "first," "second," "primary," or "secondary" used herein merely modify various components without regard to order and / or importance and distinguish one component from other components, but do not limit the components. For example, a first user device and a second user device refer to different user devices regardless of order or importance. For example, a first component may be named a second component, and similarly, the second component may be named in place of the first component, without departing from the scope of the invention as described herein.

[0019] In this specification, the terms "have," "may have," "include," or "may include" indicate the presence of a given feature (e.g., a value, a function, an operation, or a component such as a part) and do not exclude the presence of additional features.

[0020] When a component (e.g., a first component) is referred to as being "operatively or communicatively coupled with" or "connected to" another component (e.g., a second component), it should be understood that the component is directly coupled to the other component or is coupled through another component (e.g., a third component). On the other hand, when a component (e.g., a first component) is referred to as being "directly coupled with" or "directly connected to" the other component (e.g., a second component), it should be understood that there is no other component (e.g., a third component) between the component and the other component.

[0021] As used herein, the expression "configured to" is used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of," depending on the context.

[0022] The term "configured to" does not necessarily mean "specifically designed to" hardware. Instead, depending on the context, the phrase "device configured to" may mean that the device, together with other devices or components, is capable of performing the operations. For example, the phrase "a processor configured to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing the operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform the operations by executing one or more software programs stored in a memory device. Terms used herein are used solely to describe particular embodiments and are not intended to limit the scope of other embodiments. The singular includes the plural unless the context clearly dictates otherwise. Terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art described herein. Among the terms used in this specification, terms defined in a general dictionary are interpreted as meanings identical to or similar to the meanings they have in the context of the related art, and unless explicitly defined in this specification, they are not interpreted as idealized or overly formal. In some cases, even terms defined in this specification cannot be interpreted as excluding embodiments of the present invention.

[0023] As used herein, expressions such as "A or B," "at least one of A and / or B," or "one or more of A and / or B" include all possible combinations of the items listed together. For example, "A or B," "at least one of A and B," or "at least one of A or B" refer to all cases where (1) at least one A is included, (2) at least one B is included, or (3) both at least one A and at least one B are included. Furthermore, in describing components of embodiments of the present invention, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, or C," and "at least one of A, B, C, or some combination thereof" each include any one of the items listed together with the phrase, or all possible combinations thereof. In particular, a phrase such as "at least one of A, B, C, or some combination thereof" includes A, B, or C, or combinations thereof such as AB or ABC.

[0024] Hereinafter, an embodiment of the present invention will be described in detail with reference to FIGS.

[0025] FIG. 1 is a diagram showing a robot control device according to an embodiment of the present invention.

[0026] The robot controller 100 according to this embodiment includes a processor 110 and a memory 120 that includes an instruction code 122 .

[0027] The robot controller 100 controls a robot by inferring the user's intention based on the user's behavior and the situation in which the behavior occurred. For example, the robot controller 100 receives user behavior data from the robot, including the user's behavior and the situation in which the behavior occurred. Here, the user includes a human recognized by the robot through a perception module. The user's intention includes the user's intention and predicted behavior (e.g., secondary behavior following the primary behavior) based on the user's behavior (e.g., primary behavior) recognized by the robot.

[0028] The robot controller 100 applies the user behavior data to a knowledge graph. Specifically, the robot controller 100 utilizes the knowledge graph to infer or acquire a user intention from the user behavior data. Then, the robot controller 100 acquires user intention data related to the intention of the behavior by applying information related to the user behavior data, among the information included in the knowledge graph, to a rule generation model. The robot controller 100 controls the robot based on the user intention data so that the robot performs a target task related to the user's intention. The knowledge graph represents a knowledge base that can be expressed using a graphical description that can be visually represented. In the robot controller 100, the knowledge graph includes information in the form of at least one of nodes, knowledge, clusters, themes, subthemes, keywords, or any combination thereof.

[0029] As will be described later, the robot controller 100 applies instances included in the knowledge graph to a rule generation model. The robot controller 100 acquires user intention data from the output of the rule generation model. The robot controller 100 also updates the rule generation model to acquire additional user intention data. The robot controller 100 acquires the additional user intention data from the output of the updated rule generation model. The robot controller 100 determines a predicted behavior based on the user intention data and the additional user intention data. The robot controller 100 controls the robot so that the robot performs a target task related to the determined predicted behavior.

[0030] The processor 110 executes software to control at least one other component (e.g., a hardware or software component) connected to the processor 110. The processor 110 also performs various other data processing or calculations. For example, the processor 110 stores user behavior data, user intention data, etc. in the memory 120.

[0031] For reference, the processor 110 executes all operations performed by the robot controller 100. Therefore, for convenience of explanation, the operations performed by the robot controller 100 will be mainly described as operations executed by the processor 110 in this specification. Also, for convenience of explanation, the processor 110 will mainly be described as a single processor in this specification, but is not limited thereto. For example, the robot controller 100 includes at least one or more processors. Each of the at least one or more processors executes all operations related to the operation of controlling the robot by inferring the user's intention from the user's actions and the situation in which the actions occurred.

[0032] The memory 120 temporarily and / or permanently stores various data and / or information required to execute an operation to control the robot by inferring the user's intention based on the user's actions and the circumstances in which the actions occurred, such as user action data, user intention data, knowledge graphs, and rule generation models.

[0033] The robot controller 100 further includes a communication unit (not shown). For example, the communication unit supports communication between the robot controller 100 and the robot. For example, the communication unit includes one or more components that enable communication between the robot controller 100 and the robot. For example, the communication unit includes a short-range wireless communication unit, a microphone, etc. Here, short-range communication technologies include wireless LAN (Wi-Fi), Bluetooth (registered trademark), ZigBee (registered trademark), WFD (Wi-Fi Direct), UWB (ultra-wideband), infrared data association (IrDA), Bluetooth (registered trademark) Low Energy (BLE), near field communication (NFC), etc., but are not limited thereto.

[0034] FIG. 2 is a flowchart illustrating a robot control method according to an embodiment of the present invention.

[0035] In step 210, a robot control device (e.g., the robot control device 100 of FIG. 1) according to this embodiment applies event information related to a user's behavior identified in user behavior data received from the robot and context information related to the time and space where the behavior occurred to a knowledge graph formed by the relationship between an event instance related to the event information and a context instance related to the context information. For example, if the robot recognizes a user walking away, the user behavior data includes event information related to the user's walking behavior. Furthermore, the user behavior data includes context information related to the time and space where the user's walking behavior occurred. Then, the robot control device generates an event instance from the event information and a context instance from the context information.

[0036] Data related to instances and stored in the knowledge graph is expressed as classes and instances. For example, when a robot recognizes a walking user at a target location at a target time, the robot controller generates a user instance, a target location instance, a target time instance, and a walking behavior instance. Here, the event instance includes the walking behavior instance. The context instance includes a user instance, a target location instance, and a target time instance. Furthermore, the walking behavior instance inherits from the behavior class. The user instance inherits from the user class. The target location instance inherits from the location class. The target time instance inherits from the time class.

[0037] In step 220, the robot control device acquires user intention data related to the intention of the behavior by applying event instances among the instances included in the knowledge graph to a rule generation model that generates information about the intention of the behavior. For example, the robot control device identifies event instances related to event information about the user's behavior identified in the user behavior data among the instances included in the knowledge graph. The rule generation model illustratively includes a predetermined ECA rule (Event-Condition-Action rule). For example, the ECA rule may be defined as follows:

[0038] [Table 1]

[0039] For example, an ECA rule indicates a rule that outputs a newly generated action based on the relationship between an event, a condition, and an action. Specifically, when an event is input, the robot controller judges (or determines) the conditions related to the input event through the ECA rule. The event indicates the trigger of the rule. That is, the event includes a set of user actions recognized by the robot. Events include all recognizable basic actions such as walking and standing, as well as complex user actions such as getting into a car or checking a location in detail. When the condition related to the event is determined, the robot controller outputs a predetermined action through the ECA rule. The condition indicates the specific situation in which the rule is triggered. The related condition must be met in order for the rule's operation to be triggered. The condition is embodied as a knowledge graph query that checks whether a specific triple exists.

[0040] For example, when the robot controller identifies an event instance such as "touching" in the knowledge graph and inputs it into the ECA rule, the robot controller checks the number of occurrences of the input event, the duration of the input event, and the location of the input event through the ECA rule. If the above conditions are met, the robot controller outputs an action such as "inspecting car exterior" (e.g., inspecting the exterior of the vehicle) as an action for the event such as "touching."

[0041] In step 230, the robot control device controls the robot to perform a target task related to a predicted action, which is a subsequent action of the previous action, based on the user intention data. For example, the robot control device acquires user intention data such as "inspecting the car exterior" through an ECA rule. Based on the user intention data such as "inspecting the car exterior," the robot control device determines the target task related to the predicted action, which is a subsequent action of the previous action, as "a task of approaching the user and offering to explain the vehicle."

[0042] FIG. 3 is a block diagram showing a detailed configuration of the robot control device according to one embodiment of the present invention.

[0043] The robot control device 300 according to this embodiment receives user behavior data recognized from the robot 340. Based on the received user behavior data, the robot control device 300 controls the robot 340 so that the robot 340 performs a target task related to the predicted behavior through the operations shown in Figs. 4 to 9, which will be described later.

[0044] For example, the robot control device 300 includes a first processor 310, a second processor 320, and a third processor 330. The first processor 310 converts or generates user behavior data into Resource Description Framework (RDF) data. The first processor 310 outputs user intention data related to the intention of the behavior by applying instances included in a knowledge graph to a rule generation model. The second processor 320 stores and manages the knowledge graph. The second processor 320 transmits and receives instances or knowledge graphs to and from the first processor 310 and the third processor 330. The third processor 330 updates ECA rules via a rule mining model. For convenience of explanation, the present specification describes the robot control device 300 as including at least one processor, but this is not intended to be limiting. For example, the robot control device 300 performs all of the operations of the first processor 310 to the third processor 330 through a single processor.

[0045] FIG. 4 is a diagram showing an example of user behavior data received from a robot in a robot control device according to an embodiment of the present invention.

[0046] The robot control device 400 according to this embodiment receives recognized user behavior data 410 from the robot. The robot control device 400 converts and / or generates the user behavior data 410 into RDF data via a first processor.

[0047] Specifically, the robot acquires information about the robot's external environment and the user using a camera, microphone, touch sensor, etc. The robot recognizes or recognizes basic user actions based on the acquired information. Exemplary user actions include standing, sitting, touching, opening, closing, walking, etc. The user action data 410 represents unstructured data including event information about the user's actions and context information about the time and space in which the actions occurred.

[0048] The user behavior data 410 is exemplarily stored in the robot control device 400 as data 420. For example, the data 420 includes at least one of the user's name, location, occurrence time, behavior type, or any combination thereof. Since such data 420 is unstructured data, it is converted and / or generated into RDF data by the first processor.

[0049] FIG. 5 is a diagram showing an example of RDF data generated from user behavior data in a robot control device according to an embodiment of the present invention.

[0050] The robot control device 500 according to this embodiment converts user behavior data into RDF (Resource Description Framework) data 510. Specifically, the robot control device 500 converts the user behavior data, which is unstructured data, into RDF data 510, which is structured data for storage in a knowledge graph.

[0051] The robot controller 500 generates RDF data 510, which is a data format compatible with the ontology of a knowledge graph and includes relations between event information and context information. For example, the RDF data 510 represents data including subject entities, object entities, and predicates connecting these. The robot controller 500 converts the RDF data 510 into event instances and context instances and stores them in the knowledge graph. For reference, a knowledge graph represents a graph formed by the connected relationships between instances. For convenience of explanation, this specification will describe an example of the RDF data 510 as a triple 520.

[0052] FIG. 6 is a diagram showing an example of instances stored in a knowledge graph in a robot control device according to an embodiment of the present invention.

[0053] The robot control device 600 according to this embodiment stores RDF data, which is data in triple format (subject, relation, object) input according to ontology, in a knowledge graph instance 610. The robot control device 600 builds a knowledge graph centered on event instances and stores and utilizes the knowledge graph to enable queries. For example, referring to FIG. 6, a graph 620 includes an Exhibition Area context instance and a Time Interval context instance that express spatiotemporal information in a Walking event instance. Instances are inherited from classes, which will be described in detail below.

[0054] For example, the definition of a class in the ontology of the knowledge graph utilized by the robot control device 600 is as follows:

[0055] [Table 2]

[0056] The ontology of the knowledge graph forms a domain-independent hierarchy to provide a basic structure. Illustratively, the top of the hierarchy includes an event class, an object class, a location class, and a time interval class.

[0057] The Event class includes two subclasses. For example, the Event class includes an Action class and an Intention class. The Action class includes a Composite Action, which is a composite action that integrates a sequence of actions and context information different from the individual actions corresponding to the individual actions. The PrimitiveActivity class is further classified into subclasses of Communication, Action, and Posture. The CompositeActivity class includes a ContainmentChange subclass that indicates a change in containment relationship, and its subclasses include GettingIn and GettingOut. Another subclass of CompositeActivity includes Inspecting, which indicates a detailed investigation of a specific object or place. This class is further classified into InspectingPlace and InspectingObject depending on whether it is a detailed inspection, focusing on a place, or focusing on an object.

[0058] The Agent class represents an entity that will be an actor in an event. This class again contains two subclasses: a Human class and a Robot class. The Person class represents customers and staff at a car showroom, while the Robot class represents a service robot. The PhysicalObject class represents an entity that will participate in an event. This class is divided into various subclasses: Car, Component, Device, Furniture, InformationMap. The Component class has a subclass called CarComponent, which has subclasses called CarDoor and CarTrunk, and a RobotComponent.

[0059] A more specific definition of the event class of the knowledge graph utilized by the robot control device 600 is as follows:

[0060] [Table 3]

[0061] The event class includes an extended hierarchical structure of the PrimitiveActivity class, such as request and response, which are subclasses of communication; open, close, touch, and walk, which are subclasses of action; and sit and stand, which are subclasses of posture.

[0062] As described above, the robot control device 600 inherits or generates instances from each class based on the predefined class configuration. Referring to the graph 620, the w1 event instance represents an instance inherited from the walking class, and the person1 instance represents an instance inherited from the person class. In this way, the robot control device 600 generates or manages a knowledge graph based on the relationships between instances inherited from each class.

[0063] FIG. 7 is a diagram illustrating an example of acquiring user intention data through an ECA rule in a robot control device according to an embodiment of the present invention.

[0064] The robot controller 700 according to this embodiment identifies a predetermined ECA rule 710 from the rule generation model based on applying an event instance to the rule generation model. The robot controller 700 acquires a condition related to the event instance by applying the ECA rule to the event instance. Exemplarily, the robot controller 700 acquires a condition corresponding to the applied event instance. The robot controller 700 acquires user intention data by applying a context instance paired with the event instance to the condition.

[0065] Specifically, the robot control device 700 applies the context instance to the conditions of the ECA rule 710. The context instance includes the location, time, and cause of the event instance, and is therefore applied to the conditions of the ECA rule 710. In this way, the robot control device 700 uses not only the event instance but also the context instance for intention inference through the ECA rule 710, thereby enabling more sophisticated inference than inference that considers only the event instance.

[0066] The robot controller 700 acquires user intention data based on applying the event instance and the context instance to the ECA rule 710. Referring to Figure 7, the user intention data includes a vehicle exterior inspection event such as "InspectingCarExterior." Accordingly, the robot controller 700 outputs a vehicle exterior inspection event such as "InspectingCarExterior" by utilizing the context instance from the event instance (and simultaneously, the user's action) such as "touching."

[0067] The ECA rules 710 are illustratively stored in the robot controller 700 as data 720. For example, the data 720 includes an input of an event instance that is a trigger, a condition corresponding to the input event instance, and a final output resulting from checking the condition.

[0068] FIG. 8 is a diagram showing an example of the output of the rule mining model in the robot control device according to one embodiment of the present invention.

[0069] The robot control device 800 according to this embodiment identifies, from the rule generation model, a neural compositional rule learning (NCRL)-based rule mining model that generates user intention data based on event instances among the instances included in the knowledge graph, regardless of context instances. The robot control device 800 updates the ECA rules based on the output 810 of the rule mining model and the knowledge graph.

[0070] A rule mining model is a model that, when at least one event instance (e.g., a touching event instance) is input, acquires and outputs user intention data (e.g., a gettingout event instance) corresponding to the input data. Unlike ECA rules, this rule mining model omits context instances and outputs user intention data based only on event instances.

[0071] The robot controller 800 acquires action sequence data related to the event instance from the rule mining model. The robot controller 800 acquires at least one candidate context instance related to the action sequence data from the knowledge graph. For example, the robot controller 800 acquires action sequence data including a gettingout event instance corresponding to inspectingcarexterior, inspectingcarinterior, touching, and opening event instances as action sequence data related to the event instance from the rule mining model. The robot controller 800 acquires candidate context instances paired with the event instance included in the action sequence data from the knowledge graph. Specifically, the event instance is stored in the knowledge graph in a form paired with the context instance. A method for updating the ECA rule using the paired candidate context instance will be described in detail below with reference to FIG. 9.

[0072] The robot control device 800 updates the ECA rules based on the behavior sequence data and at least one or more candidate context instances.

[0073] FIG. 9 is a diagram showing an example of an operation for updating the ECA rule in the robot control device according to one embodiment of the present invention.

[0074] A robot control device according to this embodiment (e.g., the robot control device 100 in FIG. 1) acquires at least one or more behavior sequence data from a rule mining model. The robot control device identifies target behavior sequence data 910 from the at least one or more behavior sequence data. For example, if there are event instances of opening, touching, and touching, the target behavior sequence data 910 indicates data including an inspectingcarInterior event instance corresponding to the above-mentioned event instances.

[0075] Based on identifying the target behavior sequence data 910, the robot control device acquires at least one or more candidate context instances related to the target behavior sequence data 910 from the knowledge graph. Illustratively, the clustering result 920 includes at least one or more candidate context instances.

[0076] The robot control device identifies a first candidate context instance and a second candidate context instance different from the first candidate context instance from the at least one candidate context instance. The robot control device determines a first count, which is the number of objects that satisfy the combination of a location (e.g., car-car-car-car) included in the first candidate context instance and the target behavior sequence data 910. The robot control device determines a second count, which is the number of objects that satisfy the combination of a location (e.g., drive-on_desk-car-car-car) included in the second candidate context instance and the target behavior sequence data 910.

[0077] Illustratively, the clustering result 920 shows the location and time at which an event occurred for each of at least one or more candidate context instances related to the target behavior sequence data 910 .

[0078] The robot control device determines one of the first candidate context instance or the second candidate context instance as a target context instance for the behavior sequence data based on a comparison between the first count and the second count. Referring to Figure 9, the robot control device determines the first candidate context instance for the first count with the higher number out of the first count (e.g., 28) and the second count (e.g., 3) as the target context instance. However, the method for determining the target context instance is not limited thereto. For example, the expert 930 identifies the clustering result 920 and determines one of the candidate context instances included in the clustering result 920 as the target context instance.

[0079] Based on the determination of the target context instance (e.g., the context instance having a count of 28 in FIG. 9) for the target behavior sequence data 910, the robot control device updates the ECA rule 940 by inserting the relationship between the behavior sequence data and the target context instance into the ECA rule 940.

[0080] Based on the update of the ECA rule 940, the robot control device applies the instance of the event information to the ECA rule 940 to acquire additional conditions related to the instance of the event information, and applies the instance of the context information paired with the event information to the additional conditions to acquire additional user intention data different from the user intention data. The robot control device determines a target task related to predicted behavior based on the user intention data (i.e., the action inferred from the ECA rule before the update) and the additional user intention data (i.e., the action inferred from the ECA rule after the update), and controls the robot so that the robot performs the target task.

[0081] FIG. 10 is a diagram illustrating a computing system relating to a robot control device or a robot control method according to an embodiment of the present invention.

[0082] Referring to FIG. 10, a computing system 1000 relating to a robot control device or a robot control method includes at least one processor 1100, memory 1300, a user interface input device 1400, a user interface output device 1500, storage 1600, and a network interface 1700, all connected via a bus 1200.

[0083] The processor 1100 is a central processing unit (CPU) or a semiconductor device that executes processing based on instructions stored in the memory 1300 and / or the storage 1600. The memory 1300 and the storage 1600 may include various types of volatile or non-volatile recording media. For example, the memory 1300 may include a read only memory (ROM) and a random access memory (RAM).

[0084] Thus, the steps of a method or algorithm described in the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by processor 1100, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, or on a recording medium such as a hard disk, a removable disk, or a CD-ROM (i.e., memory 1300 and / or storage 1600).

[0085] An exemplary storage medium is coupled to processor 1100, such that processor 1100 reads information from and records information on the storage medium. Alternatively, the storage medium may be integral to processor 1100. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a user terminal. Alternatively, the processor and the storage medium may reside as discrete components in a user terminal.

[0086] The above description is merely an illustrative example of the technical concept of the present invention, and various modifications and variations are possible by those skilled in the art without departing from the essential characteristics of the present invention.

[0087] The above-described embodiments may be implemented using hardware components, software components, and / or a combination of hardware and software components. For example, the devices, methods, and components described herein may be implemented using a general-purpose computer or a special-purpose computer, such as a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field programmable gate array (FPGA), programmable logic unit (PLU), microprocessor, or other device that executes and responds to instructions. The processing device executes an operating system (OS) and software applications that run on the operating system. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For ease of understanding, a single processing device may be described. However, those skilled in the art will recognize that a processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing device may include multiple processors or one processor and one controller. Other processing configurations, such as a parallel processor, are also possible.

[0088] Software includes computer programs, code, instructions, or a combination of one or more of these, which configure a processing device to operate as desired or, independently or collectively, instruct the processing device. The software and / or data may be permanently or temporarily embodied in some type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed across network-coupled computing systems and stored or executed in a distributed manner. The software and data may be stored on computer-readable storage media.

[0089] The method according to the present invention may be embodied in the form of program instructions executed by various computer means and recorded on a computer-readable medium. The computer-readable medium may include, alone or in combination, program instructions, data files, data structures, etc. The program instructions recorded on the medium may be specially designed and constructed for the present invention or may be known and available to those skilled in the art of computer software. Examples of computer-readable mediums include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include not only machine language code, such as that produced by a compiler, but also high-level language code that is executed by a computer using an interpreter, etc.

[0090] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, or vice versa.

[0091] Although the embodiments have been described above with reference to the drawings, those skilled in the art can apply various technical modifications and variations thereto. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than described, or may be replaced or substituted by other components or equivalents, and still achieve suitable results.

[0092] Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the claims.

[0093] Therefore, the embodiments disclosed in this specification are for illustrative purposes only and are not intended to limit the technical idea of ​​the present invention. The scope of protection of the present invention should be interpreted according to the claims, and all technical ideas within the scope of the claims should be interpreted as being included in the scope of the present invention. [Explanation of symbols]

[0094] 100, 300, 400, 500, 600, 700, 800 Robot control device 110, 1100 processors 120, 1300 memory 122 Command word 310, 320, 330 1st to 3rd processors 340 Robot 410 User Behavior Data 420, 720 data 510 Resource Description Framework (RDF) Data 520 Triple 610 Knowledge Graph Instances 620 graphs 710, 940 ECA Regulations 810 Rule Mining Model Output 910 Target behavior sequence data 920 Clustering Results 930 Experts 1000 Computing Systems 1200 Bus 1400 User Interface Input Device 1500 User interface output device 1600 Storage 1700 network interface

Claims

1. a memory storing computer-executable instructions; and at least one processor that accesses the memory and executes the instructions; The at least one processor Applying event information related to a user's behavior identified from the user behavior data perceived by the robot and context information related to the time and space in which the behavior occurred to a knowledge graph formed by a relationship between an event instance related to the event information and a context instance related to the context information; applying the event instance among the instances included in the knowledge graph to a rule generation model that generates information about the intent of the behavior to obtain user intent data about the intent of the behavior; A robot control device characterized by controlling the robot so that the robot performs a target task related to a predicted action that is a subsequent action of the action based on the user intention data.

2. The at least one processor Identifying a predetermined Event-Condition-Action rule (ECA rule) from the rule generation model based on applying the event instance to the rule generation model; applying the event instance to the ECA rule to obtain a condition on the event instance; The robot control device according to claim 1 , wherein the user intention data is acquired by applying the context instance paired with the event instance to the condition.

3. The at least one processor identifying a Neural Compositional Rule Learning (NCRL)-based rule mining model that generates the user intention data based on the event instance among the instances included in the knowledge graph, regardless of the context instance, from the rule generation model; The robot control device according to claim 2 , wherein the ECA rules are updated based on the output of the rule mining model and the knowledge graph.

4. The at least one processor Obtaining behavior sequence data for the event instance from the rule mining model; obtaining at least one or more candidate context instances related to the behavioral sequence data from the knowledge graph; The robot control device according to claim 3 , wherein the ECA rule is updated based on the action sequence data and the at least one or more candidate context instances.

5. The at least one processor identifying a first candidate context instance and a second candidate context instance different from the at least one candidate context instance; determining a first count, which is the number of objects that satisfy a combination of the location included in the first candidate context instance and the behavioral sequence data; determining a second count, which is the number of objects that satisfy a combination of the location included in the second candidate context instance and the behavioral sequence data; The robot control device according to claim 4, wherein one of the first candidate context instance or the second candidate context instance is determined as a target context instance for the behavior sequence data based on a comparison between the first count and the second count.

6. The robot control device according to claim 5, wherein the at least one processor updates the ECA rule by inserting a relationship between the action sequence data and the target context instance into the ECA rule based on the determination of the target context instance related to the action sequence data.

7. The at least one processor applying the event information instance to the ECA rule based on the ECA rule update to obtain additional conditions related to the event information instance; The robot control device according to claim 3 , wherein additional user intention data different from the user intention data is obtained by applying an instance related to context information paired with the event information to the additional condition.

8. The at least one processor determining a target task related to the predicted behavior based on the user intention data and the additional user intention data; The robot control device according to claim 7, wherein the robot is controlled so that the robot performs the target task.

9. The at least one processor generating Resource Description Framework (RDF) data, which is a data format compatible with the ontology of the knowledge graph and includes a relationship between the event information and the context information; The robot control device according to claim 1 , wherein the RDF data is converted into the event instances and the context instances and stored in the knowledge graph.

10. A robot control method for a robot control device executed by at least one processor, comprising: applying event information regarding a user's behavior identified from the user behavior data perceived by the robot and context information regarding a time and space in which the behavior occurred to a knowledge graph formed by a relationship between an event instance regarding the event information and a context instance regarding the context information; acquiring user intention data regarding the intent of the behavior by applying the event instance among the instances included in the knowledge graph to a rule generation model that generates information regarding the intent of the behavior; and controlling the robot so that the robot performs a target task related to a predicted action that is a subsequent action of the action based on the user intention data.

11. identifying a predetermined Event-Condition-Action (ECA) rule from the rule generation model based on applying the event instance to the rule generation model; applying the event instance to the ECA rule to obtain a condition on the event instance; The robot control method of claim 10, further comprising: acquiring the user intention data by applying the context instance paired with the event instance to the condition.

12. The step of acquiring user intention data includes: identifying, from the rule generation model, a Neural Compositional Rule Learning (NCRL)-based rule mining model that generates the user intention data based on the event instances among the instances included in the knowledge graph, regardless of the context instances; and updating the ECA rules based on the output of the rule mining model and the knowledge graph.

13. The step of acquiring user intention data includes: obtaining behavior sequence data for the event instance from the rule mining model; obtaining at least one candidate context instance related to the behavior sequence data from the knowledge graph; and updating the ECA rule based on the behavior sequence data and the at least one candidate context instance.

14. The step of acquiring user intention data includes: identifying a first candidate context instance and a second candidate context instance different from the at least one candidate context instance; determining a first count, which is the number of objects that satisfy a combination of the location included in the first candidate context instance and the activity sequence data; determining a second count, which is the number of objects that satisfy a combination of the location and the activity sequence data included in the second candidate context instance; and determining one of the first candidate context instance or the second candidate context instance as a target context instance for the behavior sequence data based on a comparison of the first count and the second count.

15. 15. The robot control method of claim 14, wherein the acquiring of the user intention data includes updating the ECA rule by inserting a relationship between the action sequence data and the target context instance into the ECA rule based on the determination of the target context instance related to the action sequence data.

16. The step of acquiring user intention data includes: applying the event information instance to the ECA rule based on the ECA rule update to obtain additional conditions related to the event information instance; and applying an instance related to context information paired with the event information to the additional condition to obtain additional user intention data different from the user intention data.

17. determining a target task related to the predicted behavior based on the user intention data and the additional user intention data; 17. The method of claim 16, further comprising: controlling the robot so that the robot performs the target task.

18. generating Resource Description Framework (RDF) data, which is a data format compatible with the ontology of the knowledge graph and includes a relationship between the event information and the context information; The robot control method according to claim 10 , further comprising: converting the RDF data into the event instances and the context instances and storing the converted RDF data into the knowledge graph.

Images