Method of generating question for robot using LLM and device

Abstract

A method and apparatus for generating questions based on a large-scale language model for a robot are disclosed. The method for generating questions based on a large-scale language model for a robot comprises the steps of: recognizing a situation surrounding a robot from data regarding an environment surrounding a robot acquired using a camera or sensor to perform a task; determining one of a plurality of possible question types regarding the recognized situation; generating a question by combining the determined question type and the recognized situation; determining the validity of the generated question using a language model; and transmitting the generated question to the large-scale language model.

Description

Technology Field

[0001] The present invention relates to a method and apparatus for generating questions based on a large-scale language model for robots, and more specifically, to a method and apparatus for a robot to generate questions suitable for a problem situation on its own to complete a task. Background Technology

[0002] Conventionally, data necessary for training was collected and used to train the robot. However, when training a robot using conventional methods, there is a problem in that it is difficult to secure sufficient data to respond to various environmental changes.

[0003] Recently, research utilizing Large Language Models (LLMs) to enable robots to understand natural language commands and perform actions, as well as research aimed at facilitating natural conversation between robots and humans, is actively underway.

[0004] Technology is needed to respond to various environmental changes by using large-scale language models for robot learning. Prior art literature

[0005] Korean Registered Patent Publication No. 10-2732445 (2024.11.15) The problem to be solved

[0006] The technical problem that the present invention aims to solve is to provide a method and apparatus for generating questions based on a large-scale language model for robots, enabling the robot to learn autonomously and solve problem situations. means of solving the problem

[0007] A method for generating a question based on a large-scale language model for a robot, performed by a processor according to an embodiment of the present invention, comprises the steps of: recognizing a situation around a robot from data regarding an environment around a robot acquired using a camera or sensor to perform a task; determining one of a plurality of possible question types regarding the recognized situation; generating a question by combining the determined question type and the recognized situation; determining the validity of the generated question using a language model; and transmitting the generated question to the large-scale language model.

[0008] The step of recognizing the situation around the robot includes the step of recognizing objects by applying the acquired data to a neural network, the step of classifying the robot task into action sequences, and the step of determining whether any one of the sub-actions of the action sequences has been successfully performed using the recognized objects.

[0009] The step of determining a question type suitable for the above situation includes the step of calculating the probability that each of a plurality of question types will be output based on the recognized situation, and the step of determining the question type corresponding to the probability having the highest value among the calculated values ​​as the final question type, wherein the plurality of question types include an information acquisition type, a problem solving type, an action selection type, and an information verification type.

[0010] The step of generating the above question includes a step of determining whether the above question is generated in multiple numbers, a step of calculating the similarity between the above task and the above multiple questions if the above question is generated in multiple numbers, and a step of generating the question corresponding to the similarity having the highest value among the calculated values ​​as the final question.

[0011] The step of calculating the similarity between the above task and the above plurality of questions includes the step of generating a first vector by mapping the above task into a vector space, the step of generating a plurality of second vectors by mapping each of the above plurality of questions into a vector space, and the step of calculating the cosine similarity between the first vector and the plurality of second vectors.

[0012] The step of determining the validity of the generated question is a step of using an artificial intelligence algorithm, wherein the artificial intelligence algorithm embeds the generated question into BERT, transforms the embedded CLS token using a fully connected layer, and transforms the transformed value using a softmax function, and is trained based on the transformed value such that it includes a first node representing the probability that the generated question is valid and a second node representing the probability that the generated question is invalid.

[0013] An apparatus according to an embodiment of the present invention comprises a processor that executes large-scale language model-based question generation commands for a robot and a memory that stores said commands, wherein the commands are implemented to recognize a situation around a robot from data regarding an environment around a robot acquired using a camera or sensor to perform a task, determine one of a plurality of possible question types regarding said recognized situation, generate a question by combining said determined question type and said recognized situation, determine the validity of said generated question using a language model, and transmit said generated question to said large-scale language model. Effects of the invention

[0014] The large-scale language model-based question generation method for robots according to an embodiment of the present invention has the effect of enabling the robot to learn on its own by generating questions and solving problems to complete a task. Brief explanation of the drawing

[0015] Detailed descriptions of each drawing are provided to help to more fully understand the drawings cited in the detailed description of the present invention. FIG. 1 shows a block diagram of a robot according to an embodiment of the present invention. FIG. 2 shows a flowchart of a large-scale language model-based question generation method for a robot according to an embodiment of the present invention. Figure 3 shows a flowchart of operations performed by commands included in the mission conversion module illustrated in Figure 1. FIG. 4 is a table exemplifying the actions generated by the commands included in the mission conversion module illustrated in FIG. 1. Figure 5 shows a flowchart of operations performed by commands included in the situation awareness module illustrated in Figure 1. Figure 6 shows a flowchart of operations performed by commands included in the question type determination module and question generation module illustrated in Figure 1. Figure 7 shows the structure of a neural network executed by a command included in the question validation module illustrated in Figure 1. Specific details for implementing the invention

[0016] Specific structural or functional descriptions regarding embodiments according to the concept of the present invention disclosed herein are provided merely for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described herein.

[0017] Embodiments according to the concept of the present invention may be subject to various modifications and may take various forms; therefore, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all modifications, equivalents, or substitutions that fall within the spirit and scope of the present invention.

[0018] Terms such as "first" or "second" may be used to describe various components, but said components should not be limited by said terms. For the sole purpose of distinguishing one component from another, for example, without departing from the scope of rights according to the concept of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0019] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. Conversely, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions describing the relationship between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0020] The terms used herein are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to indicate the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0021] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0022] Hereinafter, the present invention will be described in detail by explaining preferred embodiments of the present invention with reference to the attached drawings.

[0023] FIG. 1 shows a block diagram of a robot according to an embodiment of the present invention.

[0024] Referring to FIG. 1, the robot (1) may include a large-scale language model-based question generation unit (2) for robots. The large-scale language model-based question generation unit (2) for robots refers to a unit in which the robot (1) can generate questions on its own and query a large-scale language model (LLM) to solve a problem situation. A problem situation refers to a situation in which the robot (1) cannot perform a task. For example, the large-scale language model refers to GPT-3, LaMDA, or BERT.

[0025] The robot (1) may include a drive module (30). The robot (1) may perform a task through the drive module (30). The drive module (30) may include a configuration necessary for the robot (1) to perform a task.

[0026] According to an embodiment, the robot (1) can be utilized in various fields. According to an embodiment, if the task of the robot (1) is cleaning, the drive module (30) may include a drive motor, wheels, a rotating brush, a suction fan, a servo motor, etc. According to an embodiment, if the task of the robot (1) is serving, the drive module (30) may include a drive motor, wheels, a suspension, a tray, a gyroscope, or a gimbal, etc.

[0027] A large-scale language model-based question generation unit (2) for robots may include a sensor module (10) and a computation module (20).

[0028] The sensor module (10) may include a camera and a sensor. According to an embodiment, the sensor of the sensor module (10) may include at least one of a lidar, an ultrasonic sensor, an infrared sensor, a ToF sensor, a gyroscope sensor, an IMU, a wheel encoder, an optical sensor, a temperature sensor, a pressure sensor, a magnetic sensor, a magnetic sensor, a gravity sensor, and a microswitch, but is not limited thereto.

[0029] The computation module (20) can receive data from the sensor module (10). It can receive image data generated by a camera or sensor data generated by a sensor. The computation module (20) can recognize the situation around the robot (1) from the data received from the sensor module (10). The recognition of the situation around the robot (1) will be explained in detail later. The computation module (20) generates and processes questions based on the recognized situation around the robot (1), and can control the operation of the drive module (30) according to the processing result.

[0030] In addition, the computation module (20) can divide the task of the robot (1) into step-by-step main actions, divide each main action into one or more sub-actions, and control the operation of the drive module (30) according to either the main action or the sub-action. This will be explained in detail later.

[0031] The computation module (20) may include a processor (21) that executes large-scale language model-based question generation commands for robots, a communication submodule (22) that stores said commands, and a memory (23).

[0032] The user can communicate with the robot (1) through the communication submodule (22). The user can check the status of the robot (1) through the communication submodule (22). The user can command the robot (1) to perform a task through voice or input means (e.g., buttons or a graphic user interface).

[0033] The memory (23) may include a mission conversion submodule (100), a situation awareness submodule (200), a question type determination submodule (300), a question generation submodule (400), a question validity determination submodule (500), and a question processing submodule (600).

[0034] The mission conversion submodule (100), situation awareness submodule (200), question type determination submodule (300), question generation submodule (400), question validity determination submodule (500), and question processing submodule (600) may represent a set of commands or a program. The mission conversion submodule (100), situation awareness submodule (200), question type determination submodule (300), question generation submodule (400), question validity determination submodule (500), and question processing submodule (600) may be implemented as large-scale language model-based question generation commands for robots.

[0035] The above large-scale language model-based question generation commands for robots may include a task conversion submodule (100), a situation awareness submodule (200), a question type determination submodule (300), a question generation submodule (400), a question validity determination submodule (500), and a question processing submodule (600). The above large-scale language model-based question generation commands for robots may be divided into a task conversion submodule (100), a situation awareness submodule (200), a question type determination submodule (300), a question generation submodule (400), a question validity determination submodule (500), and a question processing submodule (600).

[0036] In FIG. 1, the computation module (20) is shown as being included in the robot (1), but according to the embodiment, the computation module (20) may be physically separated from the robot (1) and implemented outside the robot (1). In this case, the computation module (20) may be included in a computing device and may communicate with the robot (1) through a communication submodule (22). The computing device may be a smartphone, tablet PC, laptop, computer, or a device equivalent thereto.

[0037] According to the embodiment, the robot (1) can be implemented in various shapes.

[0038] FIG. 2 shows a flowchart of a large-scale language model-based question generation method for a robot according to an embodiment of the present invention.

[0039] Referring to FIGS. 1 and 2, the processor (21) can subdivide the task of the robot (1) to generate an action sequence including multiple actions and vector embedding the multiple actions (S10). This may be referred to as the task subdivision and transformation step (S10).

[0040] The processor (21) recognizes the situation around the robot (1) from data regarding the environment around the robot (1) obtained using the sensor module (10), and can perform at least one of the plurality of actions (S20). If a situation occurs in which an action cannot be performed at this step (S20), the processor (21) recognizes the current situation as a problem situation and can return the problem situation. This may be referred to as the situation recognition and action execution step (S20).

[0041] The processor (21) can determine a question type suitable for the problem situation among a plurality of question types (S30). This may be referred to as the question type determination step (S30).

[0042] The processor (21) can generate a question based on the question type determined in the previous step (S30) (S40). This may be referred to as the question generation step (S40).

[0043] The processor (21) can determine the validity of the question generated in the previous step (S40) (S50). This may be referred to as the question validity determination step (S50).

[0044] The processor (21) queries the neural network with a question determined to be valid in the previous step (S50), receives the response generated by the neural network for the query, and can perform an action to resolve the problem situation (S60). This may be referred to as the question processing step (S60). The neural network refers to the large-scale language model mentioned above.

[0045] Figure 3 shows a flowchart of operations performed by the commands included in the mission conversion module illustrated in Figure 1.

[0046] Referring to FIGS. 1 to 3, FIG. 3 shows a flowchart of operations performed by commands included in the mission conversion submodule (100) in the mission subdivision and conversion step (S10) illustrated in FIG. 2.

[0047] The mission conversion submodule (100) may include one or more commands for performing mission input. The processor (21) can input a mission by executing the commands included in the mission conversion submodule (100) (S110).

[0048] According to an embodiment, the processor (21) can input a task received from a user through a communication submodule (22). According to an embodiment, the processor (21) can input a task received through an input means (e.g., touch screen, button, graphic user interface, etc.) included in the robot (1). According to an embodiment, if the data received from the sensor module (10) satisfies a predefined condition, the processor (21) can determine a task to be performed according to a pre-set mapping table and input the determined task.

[0049] The mission conversion submodule (100) may include one or more commands for converting an input mission into a main action sequence. The processor (21) can execute the commands included in the mission conversion submodule (100) to convert the mission input in the previous step (S110) into a main action sequence (S120).

[0050] The process of converting a mission into a main action sequence may be a process of subdividing the mission into multiple main actions performed in stages. The main action sequence may include the multiple main actions. In this step (S120), the multiple main actions may be generated. A mission refers to a higher-level concept than a main action.

[0051] The mission conversion submodule (100) may include one or more commands for converting a main action sequence into a sub action sequence. The processor (21) can execute the commands included in the mission conversion module (100) to convert the main action sequence generated in the previous step (S110) into a sub action sequence (S130).

[0052] The process of converting a main action sequence into a sub-action sequence may be a process of subdividing each of the plurality of main actions into one or more sub-actions. The sub-action sequence may include the one or more sub-actions. In this step (S130), a plurality of sub-actions may be generated. A main action refers to a higher-level concept than a sub-action.

[0053] According to the embodiment in the previous step (S120, S130), the robot (1) can subdivide the mission into the plurality of main actions or subdivide each of the plurality of main actions into one or more sub-actions by using a neural network of the Transformer, LSTM, or BERT family that is learned based on demonstration data for the mission.

[0054] In addition, according to an embodiment, the robot (1) can receive the above task or the above main action expressed in natural language as input using a large-scale language model such as GPT, T5, CodeX, PaLM, and subdivide it into the above multiple main actions or the above one or more sub-actions corresponding thereto.

[0055] In addition, according to an embodiment, the robot (1) can use a neural network based on hierarchical reinforcement learning, such as Options Framework, FeUdal Networks, HI-MAP, to select the plurality of main actions corresponding to the mission and determine the one or more sub-actions for each of the plurality of main actions.

[0056] The mission conversion submodule (100) may include one or more instructions for vector embedding main actions and sub actions. The processor (21) executes the instructions included in the mission conversion submodule (100) so that a plurality of main actions and a plurality of sub actions generated in the previous steps (S120, 130) can each be vector embedded (S140).

[0057] According to an embodiment, in this step (S140), a BERT-family neural network such as Sentence BERT is used to embed a plurality of main actions and a plurality of sub-actions generated in the previous step (S120, 130), thereby generating a vector corresponding to each of the plurality of main actions and the plurality of sub-actions. The vector may include semantic information for each of the plurality of main actions and the plurality of sub-actions.

[0058] The main action and the sub action may be referred to as an action. The above action may include the above main action and the above sub action.

[0059] The vector generated by embedding the above main action may be referred to as the main action vector. The vector generated by embedding the above sub action may be referred to as the sub action vector. The above main action vector and the above sub action vector may be referred to as action vectors. The above action vector may include the above main action vector and the above sub action vector.

[0060] The mission conversion submodule (100) may include one or more commands for returning the action vector. The processor (21) may execute the commands included in the mission conversion submodule (100) to return the action vector generated in the previous step (S140) (S150). The action vector returned in this step (S150) may be referenced when executing commands of the situation awareness submodule (200).

[0061] FIG. 4 is a table exemplifying the actions generated by the commands included in the mission conversion module illustrated in FIG. 1. Figure 4 shows a table (110) created as an example when the task of the robot (1) is 'room cleaning'.

[0062] Referring to FIGS. 1 to 4, as illustrated in the table (110), the mission can be divided into a plurality of main actions, and each of the main actions can be divided into a plurality of sub-actions.

[0063] The above main action corresponds to a higher-level category, and the above sub-action may correspond to a lower-level category of the above main action. Each of the above multiple main actions may be mapped to a corresponding main action vector. Each of the above multiple sub-actions may be mapped to a corresponding sub-action vector.

[0064] The above mission can be subdivided into N main actions. According to an embodiment, the above mission can be subdivided into 'map creation', 'robot position movement', and 'garbage removal'. 'Map creation' can be referred to as the first main action, 'robot position movement' as the second main action, and 'garbage removal' as the third main action. The main action name for 'map creation' can be '1', the main action name for 'robot position movement' as '2', and the main action name for 'garbage removal' as '3'. In this case, the value of N can be 3.

[0065] Each of the multiple main actions can be subdivided into m_1, m_2, m_3 … m_N sub-actions. m_1 may be the number of sub-actions corresponding to the first main action, m_2 may be the number of sub-actions corresponding to the second main action, m_3 may be the number of sub-actions corresponding to the third main action, and m_N may be the number of sub-actions corresponding to the Nth main action.

[0066] According to an embodiment, a main action (e.g., 'map generation') may be subdivided into sub-actions (e.g., 'sensor initialization', 'movement', 'data acquisition', and 'position calculation'). The sub-action name for 'sensor initialization' may be '1-1', the sub-action name for 'movement' may be '1-2', the sub-action name for 'data acquisition' may be '1-3', and the sub-action name for 'position calculation' may be '1-4'. In this case, since 'map generation' is the first main action, the value of m_1 may be 4.

[0067] According to an embodiment, a main action (e.g., 'robot position change') may be subdivided into sub-actions (e.g., 'path setting', 'obstacle avoidance', and 'destination change'). The sub-action name for 'path setting' may be '2-1', the sub-action name for 'obstacle avoidance' may be '2-2', and the sub-action name for 'destination change' may be '2-3'. In this case, since 'robot position change' is the second main action, the value of m_2 may be 3.

[0068] According to an embodiment, a main action (e.g., 'garbage removal') may be subdivided into sub-actions (e.g., 'object detection', 'garbage verification', and 'garbage collection'). The sub-action name for 'object detection' may be '3-1', the sub-action name for 'garbage verification' may be '3-2', and the sub-action name for 'garbage collection' may be '3-3'. In this case, since 'garbage removal' is the third main action, the value of m_3 may be 3.

[0069] The total number of the above multiple sub-actions can be expressed as M. As described above, if the value of m_1 is 4, the value of m_2 is 3, and the value of m_3 is 3, the value of M can be 10.

[0070] The total number of the above actions can be calculated as N+M. As described above, if the value of N is 3 and the value of M is 10, the total number of the above actions can be 13.

[0071] Figure 5 shows a flowchart of operations performed by commands included in the situation awareness module illustrated in Figure 1.

[0072] Referring to FIGS. 1 to 5, FIG. 5 shows a flowchart of operations performed by a command included in a situation recognition submodule (200) in the situation recognition and action execution step (S20) illustrated in FIG. 2.

[0073] The situation recognition submodule (200) may include one or more commands for performing situation recognition around the robot. The processor (21) can recognize the surrounding situation by executing the commands included in the situation recognition submodule (200) (S210). Recognizing the surrounding situation means that the robot (1) performs actions such as generating a map of the surrounding situation using the sensor module (10), or detecting and recognizing objects in the surrounding situation.

[0075] According to an embodiment, at this step (S210), the processor (21) can generate a three-dimensional map to recognize the situation around the robot (1). The processor (21) can form a three-dimensional map using SLAM technology that simultaneously performs self-position estimation and map generation based on data about the environment around the robot (1) acquired using the sensor module (10). At this time, the camera, lidar sensor, or IMU of the sensor module (10) may be used to acquire data about the environment around the robot (1).

[0076] Alternatively, according to an embodiment, the processor (21) can detect and recognize objects located around the robot (1). The processor (21) can detect and recognize objects located around the robot (1) using YOLO technology, which can simultaneously detect and recognize the location and type of objects within image data based on data regarding the environment around the robot (1) obtained using the sensor module (10). Alternatively, the processor (21) can recognize objects located around the robot (1) using CNN technology, which analyzes the features of objects within image data and classifies the types of objects based on data regarding the environment around the robot (1) obtained using the sensor module (10). At this time, the camera of the sensor module (10) can be used to acquire images of the robot (1).

[0077] Alternatively, according to an embodiment, the processor (21) can detect the movement of the robot (1). The processor (21) can detect the movement of the robot (1) by using Visual Odometry (VO) technology, which estimates the direction and speed of movement of the robot based on changes in viewpoint within the image data, based on data regarding the environment around the robot (1) acquired using the sensor module (10). At this time, the camera of the sensor module (10) is used to acquire images of the surroundings of the robot (1), and the Inertial Measurement Unit (IMU) of the sensor module (10) can be used to detect the movement of the robot (1).

[0078] Alternatively, according to an embodiment, the processor (21) can detect the movement of an object located around the robot (1). The processor (21) can detect the movement of an object located around the robot (1) by using Motion Tracking technology, which analyzes the direction of movement, speed, pattern, etc. of an object within the image data based on data about the environment around the robot (1) acquired using the sensor module (120), together with YOLO technology. At this time, the camera of the sensor module (10) may be used to acquire images of the robot (1).

[0079] The situation awareness submodule (200) may include one or more commands for performing an action. The processor (21) may perform an action by executing the commands included in the situation awareness submodule (200) (S220). In this step (S220), the surrounding situation of the robot (1) recognized in the previous step (S210) may be utilized. The action may refer to either the main action or the sub action.

[0080] The situation awareness submodule (200) may include one or more commands for determining whether an action is successful. The processor (21) can execute the commands included in the situation awareness submodule (200) to determine whether an action performed in the previous step (S220) is successful (S230).

[0081] According to an embodiment, in order to determine whether the action in this step (S230) is successful, the processor (21) can determine the situation around the robot (1) using the sensor module (10) or stored data.

[0082] For example, if the above action is 'moving to a destination', the processor (21) can collect information around the robot (1) using the camera, lidar, or IMU of the sensor module (10), and compare the collected information with the three-dimensional map data of the stored data to estimate the current position of the robot (1) and determine whether the robot (1) has reached the destination. If the current position of the robot (1) matches the destination, the processor (21) can determine that the 'moving to a destination' action has been successfully performed. If the current position of the robot (1) does not match the destination, the processor (21) can determine that the 'moving to a destination' action has not been successfully performed.

[0083] For example, if the above action is 'garbage collection', the processor (21) can collect information about garbage around the robot (1) using the camera of the sensor module (10) and determine whether the garbage has been collected by comparing the image before the 'garbage collection' action and the image after the 'garbage collection' action. If the garbage exists in the image before the 'garbage collection' action but does not exist in the image after the 'garbage collection' action, the processor (21) can determine that the 'garbage collection' action was successfully performed. If the garbage exists in the image before the 'garbage collection' action and also exists in the image after the 'garbage collection' action, the processor (21) can determine that the 'garbage collection' action was not successfully performed. The presence of garbage can be determined by comparing the pixels of the image or by applying the image to a neural network.

[0084] In this step (S230), if the processor (21) determines that the action was successfully performed, it may execute the next step (S240).

[0085] At this stage (S230), if the processor (21) determines that the action was not successfully performed, it may set the current situation as a problem situation and return problem situation data (S250).

[0086] According to an embodiment, at this step (S230), the problem situation data may be stored in the following format.

[0087] Situation Info, Action Failure Status, = Problem_Situation_Detect()

[0088] 'Problem_Situation_Detect()' is a problem situation detection function that returns situation information, whether an action failed, and the name of the action.

[0089] The above situation information may include data of an object located around the robot (1). The data of the object may include the name (or type) of the object, the location of the object, and information on the movement of the object.

[0090] According to an embodiment, the robot (1) may fail the 'destination move' sub-action due to the chair, which is an object. In this case, the situation information may include data of the chair.

[0091] The data of the above chair may be expressed as '(“Chair”, [0.1, 1, 0], [0, 0, 0])'. “Chair” may correspond to the name of the above object. '[0.1, 1, 0]' corresponds to the position of the above object and may indicate that the above chair is located at a position of 0.1m in the horizontal direction, 1m in the front-back direction, and 0m in the vertical direction relative to the position of the robot (1). '[0, 0, 0]' corresponds to the movement of the above object and may indicate that the above object has no movement in the horizontal direction, the front-back direction, and the vertical direction.

[0092] The direction in which the robot (1) basically moves can be defined as forward, and the opposite direction of the forward can be defined as backward. In the forward and backward directions, the forward can be defined as a positive direction and the backward as a negative direction. The horizontal direction is a direction perpendicular to the forward and backward directions in a plane perpendicular to the direction of gravity, and the left direction can be defined as a negative direction and the right direction as a positive direction. The vertical direction is based on the direction of gravity, and the downward direction can be defined as a negative direction and the upward direction as a positive direction.

[0093] The above action failure status may include data set to one of two values ​​for a specific action. According to an embodiment, if the specific action fails, the value of the action failure status may be set to 'YES'. According to an embodiment, if the specific action succeeds, the value of the action failure status may be set to 'NO'. According to an embodiment, the value of the action failure status may not be set before the specific action is performed.

[0094] The above action name may be set as the main action name or the sub action name depending on the type of the above specific action. For example, if the above specific action is 'Move to Destination', the above action name may be set as '2-3', which is the sub action name of 'Move to Destination'.

[0095] The situation awareness submodule (200) may include one or more commands for determining whether the mission has been completed. The processor (21) can determine whether the mission has been completed by executing the commands included in the situation awareness module (200) (S240).

[0096] If it is determined at this stage (S240) that the above mission has been completed, the processing process regarding the above mission may be terminated.

[0097] If it is determined that the above mission is not completed at this stage (S240), the processor (21) may execute the previous stage (S210) to perform the action corresponding to the next stage of the last action performed, starting from the stage of recognizing the surrounding situation of the robot (1).

[0098] Unlike what is shown in FIG. 5, if the above mission is not completed, the processor (21) may re-execute the previous step (S220) to perform an action corresponding to the next step of the last action performed, but may use the surrounding conditions of the robot (1) recognized in the previous step (S210).

[0099] Figure 6 shows a flowchart of operations performed by commands included in the question type determination module and question generation module illustrated in Figure 1.

[0100] Referring to FIGS. 1 to 6, FIG. 6 shows a flowchart of operations performed by a command included in the question type determination submodule (300) in the question type determination step (S30) shown in FIG. 2, and a flowchart of operations performed by a command included in the question generation submodule (400) in the question generation step (S40) shown in FIG. 2.

[0101] The question type determination submodule (300) may include one or more commands for calculating the probability that each of the multiple question types will be output based on the returned problem situation. The processor (21) can execute the commands included in the question type determination submodule (300) to calculate the probability that each of the multiple question types will be output based on the returned problem situation (S310).

[0102] According to an embodiment, the plurality of question types may include information acquisition type, problem solving type, action selection type, information verification type, etc.

[0103] The above information acquisition type may be a question type for acquiring information about an object located around the robot (1). According to an embodiment, the above information acquisition type may include a question such as “What is this object?”

[0104] The above problem-solving type may be a type that asks how to solve a problem in a specific problem situation. According to an embodiment, the above problem-solving type may include a question such as “What should be done with the clothes on the floor?”

[0105] The above action selection type may be a type that asks for the most suitable alternative among the actions that the robot (1) can take to solve a problem in a specific problem situation. According to an embodiment, the above action selection type may include questions such as “Move around the cup, or move the cup onto the table?”

[0106] The above information verification type may be a type that reconfirms information known by the robot (1). According to an embodiment, the above information verification type may include a question such as “Is a piece of paper trash?”

[0107] Based on the problem situation returned above, the probability that each of the above multiple question types will be output can be calculated using the following formula.

[0108] P(Question_j | Situation_i)

[0109] 'Question_j' represents the j-th question type among the possible question types in the returned problem situation. 'Situation_i' represents the returned problem situation. 'P(Question_j | Situation_i)' is the conditional probability that 'Question_j' is selected given 'Situation_i'.

[0110] The question type determination submodule (300) may include one or more commands to determine the question type corresponding to the probability having the highest value as the final question type. The processor (21) may execute the commands included in the question type determination submodule (300) to determine the question type corresponding to the probability having the highest value among the conditional probabilities calculated in the previous step (S310) as the final question type (S320).

[0111] The question type corresponding to the probability having the highest value among the conditional probabilities calculated in the previous step (S310) can be obtained using the following formula.

[0112] argmax_j P(Question_j | Situation_i)

[0113] 'argmax_j' is an operation symbol used to find the index j at which the value of the above conditional probability is maximized. In other words, through 'argmax_j P(Question_j | Situation_i)', the 'Question_j' at which the above conditional probability is maximized given 'Situation_i' can be determined.

[0114] In this case, there may be one or more 'Question_j's that maximize the conditional probability. If there is only one 'Question_j' that maximizes the conditional probability, one 'Question_j' is determined, and if there are multiple 'Question_j's that maximize the conditional probability, multiple 'Question_j's can be determined.

[0115] According to an embodiment, based on a situation where the robot (1) fails to perform a 'destination move' sub-action by the object chair, if the question type that maximizes the conditional probability among the plurality of question types is selected, the problem-solving type and the action selection type may be selected.

[0116] The question generation submodule (400) may include one or more commands for generating a question based on a problem situation and a determined question type. The processor (21) may execute the commands included in the question generation submodule (400) to generate a question based on the returned problem situation and the question type determined in the previous step (S320) (S410).

[0117] According to the embodiment, in this step (S410), a question can be generated using a transformer-based neural network such as T5, BERT, or LLM.

[0118] According to the embodiment, a situation in which the robot (1) fails the 'destination move' sub-action due to the chair which is an object, and based on the problem-solving type above, a question such as 'what to do with the chair on the floor?' can be generated.

[0119] According to an embodiment, a situation in which the robot (1) fails the 'destination move' sub-action due to the chair which is an object, and based on the action selection type, a question such as 'whether to move while avoiding the chair, or to move the chair?' may be generated.

[0120] The question generation submodule (400) may include one or more commands for determining whether the generated questions are multiple. The processor (21) can execute the commands included in the question generation submodule (400) to determine whether the number of questions generated in the previous step (S410) is multiple (S420).

[0121] According to an embodiment, in the previous step (S410), the first neural network may output questions in the form of a list. The number of generated questions may be counted based on the number of items in the list.

[0122] According to an embodiment, if there are multiple question types selected as the final question types in the previous step (S320), the number of questions generated in the previous step (S410) may be multiple. In this step (S420), if the processor (21) determines that the number of questions generated in the previous step (S410) is multiple, it may execute the next step (S430).

[0123] In this step (S240), if the processor (21) determines that the number of questions generated in the previous step (S410) is not multiple, it can determine the question generated in the previous step (S410) as the final question and execute the next step (S460).

[0124] The question generation submodule (400) may include one or more commands for vector embedding the generated multiple questions. The processor (21) may execute the commands included in the question generation submodule (400) to vector embedding the multiple questions generated in the previous step (S410) (S430). The vector generated by embedding each of the multiple questions may be referred to as a question vector.

[0125] According to an embodiment, in this step (S430), a vector corresponding to each of the multiple questions generated in the previous step (S410) can be generated by embedding the multiple questions generated in this step (S430) using a BERT-family neural network such as Sentence BERT. The vector may include semantic information for each of the multiple questions.

[0126] The question generation submodule (400) may include one or more commands for calculating the similarity between an action vector and a plurality of question vectors. The processor (21) may execute the commands included in the question generation submodule (400) to calculate the similarity between the action vector and the plurality of question vectors generated in the previous step (S430) (S440).

[0127] In this step (S440), the similarity between the action vector and the plurality of question vectors can be calculated based on cosine similarity. Each of the action vector and the plurality of question vectors can be mapped to a vector space, and the cosine similarity between the action vector and the plurality of question vectors can be calculated. The action vector may be referred to as the first vector, and the plurality of question vectors may be referred to as the plurality of second vectors.

[0128] This step (S440) may be a step of calculating the similarity between the above task and the plurality of vectors. To this end, the similarity between the sub-action sequence, which is part of the above task, and the plurality of question vectors may be calculated. The action vector may be a vector of a plurality of sub-actions included in the sub-action sequence corresponding to the action name included in the problem situation data. The sub-action sequence may be a set of sub-actions corresponding to a common main action. In other words, the action vector may be a vector of a plurality of sub-actions corresponding to a specific main action.

[0129] According to an embodiment, if the action name is '2-3', the action vector may be a vector of multiple sub-actions corresponding to a main action whose main action name is '2'. The multiple sub-actions are '2-1', '2-2', and '2-3', and each vector can be represented as 'v_A21', 'v_A22', and 'v_A23'.

[0130] According to an embodiment, the plurality of question vectors may include a vector corresponding to the problem-solving type question and a vector corresponding to the action-selection type question. The vector corresponding to the problem-solving type question may be represented as 'v_Question_2', and the vector corresponding to the action-selection type question may be represented as 'v_Question_3'.

[0131] The cosine similarity between the above action vector and the above multiple question vectors can be calculated using the following formula.

[0132] max_Q_2 = max{cos_sim(v_Question_2, v_A2i)}, for all i

[0133] max_Q_3 = max{cos_sim(v_Question_3, v_A2i)}, for all i

[0134] 'v_A2i' is a symbol representing the vector of sub-action '2-i' corresponding to main action '2'. 'cos_sim(v_Question_2, v_A2i)' is an operation symbol for calculating the cosine similarity between the vector corresponding to the problem-solving type question and the vector of sub-action '2-i', and 'cos_sim(v_Question_3, v_A2i)' is an operation symbol for calculating the cosine similarity between the vector corresponding to the action-selection type question and the vector of sub-action '2-i'. 'max' is an operation symbol for calculating the maximum value.

[0135] 'max_Q_2 = max{cos_sim(v_Question_2, v_A2i)}, for all i' is an operation symbol for storing the maximum value of the cosine similarity between 'v_Question_2' and the sub-action vector corresponding to the main action '2' in 'max_Q_2'.

[0136] 'max_Q_3 = max{cos_sim(v_Question_3, v_A2i)}, for all i' is an operation symbol for storing the maximum value of the cosine similarity between 'v_Question_3' and the sub-action vector corresponding to the main action '2' in 'max_Q_3'.

[0137] The question generation submodule (400) may include one or more commands to determine the question corresponding to the cosine similarity having the highest value as the final question. The processor (21) may execute the commands included in the question generation submodule (400) to obtain the highest value among the cosine similarities calculated in the previous step (S440), and determine the question corresponding to the cosine similarity having the highest value as the final question (S450). This can be obtained using the following formula.

[0138] FinalQuestion = max_Q_2 > max_Q_3 : Question_2, Question_3

[0139] 'FinalQuestion = max_Q_2 > max_Q_3 : Question_2, Question_3' is an operation symbol for storing 'Question_2' in 'FinalQuestion' if 'max_Q_2 > max_Q_3' is true, and storing 'Question_3' in 'FinalQuestion' if 'max_Q_2 > max_Q_3' is false. In other words, it is an operation symbol for determining the value with the highest cosine similarity between the action vector and the plurality of question vectors as the final question.

[0140] According to an embodiment, if 'max_Q_2' is greater than 'max_Q_3', Question_2 may be determined as the final question. In other words, the problem-solving question 'What should be done with the chair on the floor?' may be determined as the final question.

[0141] The question generation submodule (400) may include one or more commands for returning the final question. The processor (21) may execute the commands included in the question generation submodule (400) to return the final question determined in the previous steps (S420, S450) (S460). The final question returned in this step (S460) may be referenced when executing commands of the question validity determination submodule (500).

[0142] Figure 7 shows the structure of a neural network executed by a command included in the question validation module illustrated in Figure 1.

[0143] Referring to FIGS. 1 through 7, the question validity determination submodule (500) may include one or more commands for determining whether the final question is a valid question. The processor (21) may determine whether the final question is a valid question through the first neural network (610).

[0144] The data input to the first neural network (610) may be the final question data (410). Depending on the embodiment, the final question data (410) may be in the form of natural language.

[0145] The first neural network (610) may have a structure based on a Transformer Model. The first neural network (610) may include a BERT module (611), a fully connected layer (612), an activation function (e.g., softmax, 613), and an output layer (614).

[0146] The BERT module (611) can receive the final question data (410) as input and generate multiple tokens. The multiple tokens are 'CLS', 'T1', 'T2', … , 'T N It may include '. Each of the above plurality of tokens may be embedded by reflecting contextual information of the final question data (410), and among them, the 'CLS' token may include contextual information for the entire final question data (410). The 'CLS' token may be used as an input to the fully connected layer (612).

[0147] The 'CLS' token can be converted into a feature vector suitable for classification by passing through a fully connected layer (612). The feature vector can be converted into a probability value by passing through an activation function (613).

[0148] The output layer (614) may include two output nodes. The output layer (615) may include a first node and a second node. The first neural network (610) may be trained such that the first node is represented as 'n1' and represents the probability that the final question data (410) is valid, and the second node is represented as 'n2' and represents the probability that the final question data (410) is invalid.

[0149] If the processor (21) determines through the first neural network (610) that the final question is a valid question, it may query the second neural network for the valid final question. The second neural network may be implemented as a large-scale language model (LLM).

[0150] The processor (21) can receive the response generated by the second neural network for the valid final question and perform an action to resolve the problem situation.

[0151] The present invention has been described with reference to an exemplary embodiment illustrated in the drawings, but this is merely illustrative, and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the appended claims. Explanation of the symbols

[0152] 1: Robot 2: System 10: Sensor Module 20: Computation Module 21: Processor 23: Memory 30: Drive module 100: Mission Transformation Module 200: Context Awareness Module 300: Question Type Determination Module 400: Question generation module 500: Question Validation Module 600: Question processing module

Claims

Claim 1 A method for generating questions based on a large-scale language model for a robot, performed by a processor, comprising: a step of recognizing a situation around a robot from data about an environment around a robot acquired using a camera or sensor to perform a task; a step of determining one of a plurality of possible question types for the recognized situation; a step of generating a question by combining the determined question type and the recognized situation; a step of determining the validity of the generated question using a language model; and a step of transmitting the generated question to the large-scale language model, wherein the step of recognizing a situation around a robot comprises: a step of recognizing objects by applying the acquired data to a neural network; a step of classifying a robot task into an action sequence; and a step of determining whether one of the sub-actions of the action sequence has been successfully performed using the recognized objects. Claim 2 delete Claim 3 A method for generating questions based on a large-scale language model for a robot, wherein the step of determining a question type suitable for the situation comprises: a step of calculating the probability that each of a plurality of question types will be output based on the recognized situation; and a step of determining the question type corresponding to the probability having the highest value among the calculated values ​​as the final question type, and wherein the plurality of question types include an information acquisition type, a problem solving type, an action selection type, and an information verification type. Claim 4 A large-scale language model-based question generation method for a robot according to claim 1, wherein the step of generating the question comprises: determining whether the question is generated in multiple numbers; if the question is generated in multiple numbers, calculating the similarity between the task and the multiple questions; and generating the question corresponding to the similarity having the highest value among the calculated values ​​as the final question. Claim 5 A large-scale language model-based question generation method for robots, wherein the step of calculating similarity between the task and the plurality of questions comprises: a step of generating a first vector by mapping the task into a vector space; a step of generating a plurality of second vectors by mapping each of the plurality of questions into a vector space; and a step of calculating cosine similarity between the first vector and the plurality of second vectors. Claim 6 A method for generating questions based on a large-scale language model for robots, wherein the step of determining the validity of the generated question is a step of using an artificial intelligence algorithm, and the artificial intelligence algorithm embeds the generated question into BERT, transforms the embedded CLS token using a fully connected layer, and transforms the transformed value using a softmax function, wherein the transformed value is learned based on a value that includes a first node representing the probability that the generated question is valid and a second node representing the probability that the generated question is invalid. Claim 7 A device comprising: a processor that executes large-scale language model-based question generation commands for a robot; and a memory that stores said commands, wherein the commands are implemented to recognize the situation around the robot from data about the environment around the robot acquired using a camera or sensor to perform a task, determine one of a plurality of possible question types regarding said recognized situation, generate a question by combining said determined question type and said recognized situation, determine the validity of said generated question using a language model, and transmit said generated question to said large-scale language model, and wherein the commands for recognizing the situation around the robot are implemented to recognize objects by applying said acquired data to a neural network, classify the robot task into an action sequence, and determine whether any one of the sub-actions in said action sequence has been successfully performed using said recognized object.

Images