Terminal device, interlocking method, and program

Abstract

To provide a terminal device, an interlocking method, and a program, capable of improving amusement in a robot simulating an organism.SOLUTION: In a terminal device 50, a display part 540 displays an image. A communication part 550 transmits information to a robot. When the display part 540 displays an image, a control part 510 transmits robot control information of causing the robot to perform a gesture corresponding to the image to the robot via the communication part 550.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a terminal device, an interlocking method, and a program. [Background technology]

[0002] There are known robots that mimic living things such as pets and humans. For example, Patent Document 1 discloses a robot communication system that can be programmed to transmit music data and action command data from a mobile phone device to a robot device so that the robot device performs actions in accordance with the music. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-044798 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the robots that imitate living creatures as described above, there is a demand for making the robots more entertaining.

[0005] The present invention has been made to solve the above-mentioned problems, and aims to provide a terminal device, a linking method, and a program that can increase the interest in robots that simulate living creatures. [Means for solving the problem]

[0006] In order to achieve the above object, one aspect of the terminal device according to the present invention is to A display means for displaying an image; A transmitting means for transmitting information to the robot; a control means for transmitting, to the robot via the transmission means, robot control information for causing the robot to perform a gesture corresponding to the image when the image is displayed on the display means; The present invention is characterized by comprising: Effect of the Invention

[0007] According to the present invention, it is possible to improve the interest of a robot that imitates a living thing. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an outline of the overall configuration of a robot system according to a first embodiment. [Diagram 2] 1 is a cross-sectional side view of a robot according to a first embodiment. [Diagram 3] 1 is a block diagram showing a configuration of a robot according to a first embodiment. [Figure 4] 1 is a block diagram showing a configuration of a terminal device according to a first embodiment. [Diagram 5] FIG. 3 is a diagram showing an example of an emotion map according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of a personality value radar chart according to the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of gesture information according to the first embodiment. [Figure 8] FIG. 4 is a first diagram showing an example of a coefficient table according to the first embodiment. [Figure 9] FIG. 2 is a second diagram illustrating an example of a coefficient table according to the first embodiment. [Figure 10] FIG. 4 is a diagram showing an example of content data according to the first embodiment. [Figure 11] FIG. 4 is a diagram showing an example of control data according to the first embodiment. [Figure 12] 4 is a flowchart showing a flow of a robot control process according to the first embodiment. [Figure 13] 5 is a flowchart showing the flow of a gesture control process according to the first embodiment. [Figure 14]4 is a sequence diagram showing the flow of an interlocking control process according to the first embodiment. FIG. [Figure 15] FIG. 11 is a diagram showing an example of content data according to the second embodiment. [Figure 16] FIG. 11 is a first diagram showing an example of cooperative control between a terminal device and a robot according to the second embodiment. [Figure 17] FIG. 11 is a second diagram showing an example of cooperative control between the terminal device and the robot according to the second embodiment. [Figure 18] FIG. 11 is a third diagram showing an example of cooperative control between the terminal device and the robot according to the second embodiment. [Figure 19] FIG. 4 is a fourth diagram showing an example of cooperative control between the terminal device and the robot according to the second embodiment. [Figure 20] FIG. 11 is a first diagram showing an example of cooperative control between a terminal device and a robot according to the third embodiment. [Figure 21] FIG. 11 is a second diagram showing an example of cooperative control between a terminal device and a robot according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals.

[0010] (Embodiment 1) 1 shows a schematic configuration of a robot system 1 according to the embodiment 1. The robot system 1 includes a robot 200 and a terminal device 50. The robot 200 is an example of the robot according to the embodiment 1.

[0011] The robot 200 according to the first embodiment includes an exterior 201, a decorative part 202, a bushy fur 203, a head 204, a connecting part 205, a body part 206, a housing 207, a touch sensor 211, an acceleration sensor 212, a microphone 213, an illuminance sensor 214, and a speaker 231 similar to those of the robot 200 disclosed in JP 2023-115370 A, and the description thereof will be omitted. The shape of the head 204 may be the shape shown in FIG. 2 or may be the shape disclosed in, for example, FIG. 2 of JP 2023-115370 A.

[0012] The robot 200 according to the first embodiment includes a twist motor 221 and an up-down motor 222 similar to those of the robot 200 disclosed in Japanese Patent Application Laid-Open No. 2023-115370, and the description thereof will be omitted. The twist motor 221 and the up-down motor 222 of the robot 200 according to the first embodiment operate in the same manner as the robot 200 disclosed in Japanese Patent Application Laid-Open No. 2023-115370.

[0013] The robot 200 includes a gyro sensor 215. With the acceleration sensor 212 and the gyro sensor 215, the robot 200 can detect a change in the posture of the robot 200 itself, and can also detect that the robot 200 has been lifted up, changed in direction, or thrown by a user.

[0014] At least a portion of the acceleration sensor 212, microphone 213, illuminance sensor 214, gyro sensor 215 and speaker 231 may be provided in the head 204, not limited to the torso 206, or in both the torso 206 and the head 204.

[0015] Next, the functional configuration of the robot 200 will be described with reference to Fig. 3. As shown in Fig. 3, the robot 200 includes a control device 100, a sensor unit 210, a driving unit 220, an output unit 230, and an operation unit 240. These units are connected to each other via a bus line BL, for example. Note that instead of the bus line BL, a wired interface such as a USB (Universal Serial Bus) cable or a wireless interface such as Bluetooth (registered trademark) may be used.

[0016] The control device 100 is a device that controls the robot 200. The control device 100 includes a control unit 110 which is an example of a control means, a storage unit 120 which is an example of a storage means, and a communication unit 130 which is an example of a communication means.

[0017] The control unit 110 includes a CPU (Central Processing Unit). The CPU is, for example, a microprocessor, and is a central processing unit that executes various processes and calculations. In the control unit 110, the CPU reads out a control program stored in a ROM, and controls the operation of the entire robot 200, which is the control unit, while using a RAM as a work memory. Although not shown, the control unit 110 also includes a clock function, a timer function, and the like, and can measure the date and time. The control unit 110 may also be called a "processor."

[0018] The storage unit 120 includes a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, etc. The storage unit 120 stores programs and data, including an OS (Operating System) and application programs, that are used by the control unit 110 to perform various processes. The storage unit 120 also stores data that is generated or acquired by the control unit 110 performing various processes.

[0019] The communication unit 130 includes a communication interface for communicating with devices external to the robot 200. For example, the communication unit 130 communicates with external devices including the terminal device 50 in accordance with well-known communication standards such as wireless LAN (Local Area Network), BLE (Bluetooth Low Energy (registered trademark)), and NFC (Near Field Communication).

[0020] The sensor unit 210 includes the above-mentioned touch sensor 211, acceleration sensor 212, gyro sensor 215, illuminance sensor 214, and microphone 213. The sensor unit 210 is an example of a detection unit that detects an external stimulus.

[0021] The touch sensor 211 includes, for example, a pressure sensor or a capacitance sensor, and detects contact with some object. Based on the detection value of the touch sensor 211, the control unit 110 can detect whether the robot 200 is being stroked or hit by the user.

[0022] The acceleration sensor 212 detects acceleration applied to the body 206 of the robot 200. The acceleration sensor 212 detects acceleration in each of the X-axis direction, the Y-axis direction, and the Z-axis direction, that is, acceleration in three axes.

[0023] For example, the acceleration sensor 212 detects gravitational acceleration when the robot 200 is stationary. The control unit 110 can detect the current posture of the robot 200 based on the gravitational acceleration detected by the acceleration sensor 212. In other words, the control unit 110 can detect whether the housing 207 of the robot 200 is tilted from the horizontal direction based on the gravitational acceleration detected by the acceleration sensor 212. In this way, the acceleration sensor 212 functions as a tilt detection means that detects the tilt of the robot 200.

[0024] Furthermore, when the user lifts or throws the robot 200, the acceleration sensor 212 detects the acceleration accompanying the movement of the robot 200 in addition to the gravitational acceleration. Therefore, the control unit 110 can detect the movement of the robot 200 by removing the gravitational acceleration component from the detection value detected by the acceleration sensor 212.

[0025] The gyro sensor 215 detects the angular velocity when rotation is applied to the body 206 of the robot 200. Specifically, the gyro sensor 215 detects the angular velocity of three-axis rotation, namely, rotation about the X-axis direction, rotation about the Y-axis direction, and rotation about the Z-axis direction. By combining the detection values ​​detected by the acceleration sensor 212 and the detection values ​​detected by the gyro sensor 215, the movement of the robot 200 can be detected with higher accuracy.

[0026] The touch sensor 211, the acceleration sensor 212, and the gyro sensor 215 detect the strength of contact, the acceleration, and the angular velocity, respectively, at synchronized timing (for example, every 0.25 seconds), and output the detection values ​​to the control unit 110.

[0027] The microphone 213 detects sounds around the robot 200. Based on the components of the sound detected by the microphone 213, the control unit 110 can detect, for example, whether the user is calling out to the robot 200 or clapping his hands.

[0028] The illuminance sensor 214 detects the illuminance of the surroundings of the robot 200. Based on the illuminance detected by the illuminance sensor 214, the control unit 110 can detect whether the surroundings of the robot 200 have become brighter or darker.

[0029] The control unit 110 acquires, as an external stimulus, detection values ​​detected by various sensors included in the sensor unit 210 via the bus line BL. The external stimulus is a stimulus that acts on the robot 200 from outside the robot 200. Examples of the external stimulus include "a loud noise was heard," "someone spoke to," "someone stroked," "someone lifted up," "someone turned upside down," "it became brighter," "it became darker," and the like.

[0030] For example, the control unit 110 obtains an external stimulus caused by "a loud noise being made" or "being spoken to" through the microphone 213, and obtains an external stimulus caused by "being stroked" through the touch sensor 211. In addition, the control unit 110 obtains an external stimulus caused by "being lifted" or "being turned upside down" through the acceleration sensor 212 and gyro sensor 215, and obtains an external stimulus caused by "it becoming brighter" or "it becoming darker" through the illuminance sensor 214.

[0031] The sensor unit 210 may include sensors other than the touch sensor 211, the acceleration sensor 212, the gyro sensor 215, and the microphone 213. By increasing the types of sensors included in the sensor unit 210, it is possible to increase the types of external stimuli that the control unit 110 can acquire.

[0032] The driving unit 220 includes a twist motor 221 and an up-down motor 222, and is driven by the control unit 110. The twist motor 221 is a servo motor for rotating the head 204 in the left-right direction (width direction) with respect to the torso unit 206, with the front-rear direction as an axis. The up-down motor 222 is a servo motor for rotating the head 204 in the up-down direction (height direction) with respect to the torso unit 206, with the left-right direction as an axis. The robot 200 can express the motion of twisting the head 204 sideways by the twist motor 221, and can express the motion of raising and lowering the head 204 by the up-down motor 222.

[0033] The output unit 230 includes a speaker 231, and when the control unit 110 inputs sound data to the output unit 230, sound is output from the speaker 231. For example, when the control unit 110 inputs data on the cry of the robot 200 to the output unit 230, the robot 200 emits a pseudo cry.

[0034] Incidentally, instead of or in addition to the speaker 231, the output unit 230 may be provided with a display such as a liquid crystal display or a light-emitting unit such as an LED (Light Emitting Diode), and emotions such as joy and sadness may be displayed on the display or expressed by the color and brightness of the emitted light.

[0035] The operation unit 240 includes operation buttons, a volume knob, etc. The operation unit 240 is an interface for accepting user operations such as turning the power on and off, adjusting the volume of the output sound, etc.

[0036] The battery 250 is a rechargeable secondary battery, and stores power used in the robot 200. The battery 250 is charged when the robot 200 moves to a charging station.

[0037] The position information acquisition unit 260 includes a position information sensor such as a Global Positioning System (GPS) and acquires current position information of the robot 200. Note that the position information acquisition unit 260 may acquire the position information of the robot 200 by a general method using wireless communication, not limited to GPS, or may acquire the position information of the robot 200 through application software of the terminal device 50.

[0038] The control unit 110 functionally comprises a state parameter acquisition unit 112 which is an example of a state parameter acquisition means, and a gesture control unit 113 which is an example of a gesture control means. In the control unit 110, the CPU reads out a program stored in the ROM into the RAM, and executes and controls the program, thereby functioning as each of these units.

[0039] The storage unit 120 also stores gesture information 121, state parameters 122, and a coefficient table .

[0040] Next, the configuration of the terminal device 50 will be described with reference to Fig. 4. The terminal device 50 is an operation terminal operated by a user. The terminal device 50 is, for example, a general-purpose information processing device such as a smartphone, a tablet terminal, or a wearable terminal. As shown in Fig. 4, the terminal device 50 includes a control unit 510, a storage unit 520, an operation unit 530, a display unit 540, and a communication unit 550.

[0041] The control unit 510 includes a CPU. In the control unit 110, the CPU reads out a control program stored in a ROM and controls the overall operation of the terminal device 50 while using a RAM as a work memory. The control unit 510 may be called a "processor."

[0042] The storage unit 520 includes a ROM, a RAM, a flash memory, etc. The storage unit 520 stores programs and data used by the control unit 510 to perform various processes. The storage unit 520 also stores data generated or acquired by the control unit 510 performing various processes.

[0043] The operation unit 530 includes an input device such as a touch panel, a touch pad, physical buttons, etc., and receives operation input from a user. The operation unit 530 is an example of a receiving unit that receives operation input from a user.

[0044] Display unit 540 includes a display device such as a liquid crystal display, and displays various images under the control of control unit 510. Display unit 540 is an example of a display means.

[0045] The communication unit 550 includes a communication interface for communicating with devices external to the terminal device 50. For example, the communication unit 550 communicates with external devices including the robot 200 in accordance with well-known communication standards such as wireless LAN, BLE (registered trademark), and NFC. The communication unit 550 is an example of a transmission means for transmitting information to the robot 200.

[0046] The control unit 510 functionally includes an interlocking control unit 513 which is an example of an interlocking control means. In the control unit 510, the CPU reads a program stored in the ROM into the RAM and executes and controls the program, thereby functioning as the interlocking control unit 513. The storage unit 520 stores content data 521.

[0047] Returning to Fig. 3, in the control device 100 of the robot 200, the state parameter acquisition unit 112 acquires state parameters 122. The state parameters 122 are parameters for expressing the state of the robot 200. Specifically, the state parameters 122 include (1) emotion parameters, (2) personality parameters, (3) remaining battery power, (4) current location, (5) current time, and (6) number of days of growth (number of days of training).

[0048] (1) Emotion parameters The emotion parameters are parameters that represent simulated emotions for the robot 200. The emotion parameters are represented by coordinates (X, Y) on the emotion map 300.

[0049] The emotion map 300 is similar to that disclosed in JP 2023-115370 A, and therefore description thereof will be omitted.

[0050] State parameter acquisition section 112 calculates an emotion change amount, which is an amount of change that increases or decreases the X and Y values ​​of emotion parameters. The emotion change amount is expressed by the following four variables. DXP: Ease of feeling at ease (the tendency for the X value on the emotion map to change in a positive direction) DXM: Tendency to become anxious (the tendency for the X value on the emotion map to change in the negative direction) DYP: Excitability (the tendency for the Y value on the emotion map to change in the positive direction) DYM: Tendency to become apathetic (the tendency for the Y value on the emotion map to change in the negative direction)

[0051] The state parameter acquisition unit 112 updates the emotion parameter by adding or subtracting a value corresponding to the external stimulus from among the emotion change amounts DXP, DXM, DYP, and DYM to the current emotion parameter. For example, when the head 204 is stroked, the simulated emotion of the robot 200 is one of relief, so the state parameter acquisition unit 112 adds DXP to the X value of the emotion parameter. Conversely, when the head 204 is hit, the simulated emotion of the robot 200 is one of anxiety, so the state parameter acquisition unit 112 subtracts DXM from the X value of the emotion parameter. It is possible to arbitrarily set what emotion change amounts correspond to various external stimuli. An example is shown below. Head 204 is stroked (feels safe): X=X+DXP Hit on the head 204 (feels anxious): X=X-DXM (These external stimuli can be detected by the touch sensor 211 on the head 204.) The torso 206 is stroked (excited): Y=Y+DYP Hitting the torso 206 (becoming lethargic): Y=Y-DYM (These external stimuli can be detected by the touch sensor 211 on the torso 206.) Being held with head up (happy): X=X+DXP and Y=Y+DYP Hanging head down (sad): X=X-DXM and Y=Y-DYM (These external stimuli can be detected by the touch sensor 211 and the acceleration sensor 212.) A gentle voice calls out to you (becomes peaceful): X=X+DXP and Y=Y-DYM Being yelled at loudly (irritated): X=X-DXM and Y=Y+DYP (These external stimuli can be detected by microphone 213.)

[0052] The sensor unit 210 acquires a plurality of different types of external stimuli using a plurality of sensors. The state parameter acquisition unit 112 derives various amounts of emotion change in response to each of the plurality of external stimuli, and sets emotion parameters in response to the derived amounts of emotion change.

[0053] The initial value of each of the emotion variation amounts DXP, DXM, DYP, DYM is 10 and can increase up to a maximum of 20. The state parameter acquisition section 112 updates each variable of the emotion variation amounts DXP, DXM, DYP, DYM in response to an external stimuli detected by the sensor section 210.

[0054] Specifically, state parameter acquisition section 112 adds 1 to DXP if the X value of the emotion parameter is set to the maximum value of emotion map 300 even once in a day, and adds 1 to DYP if the Y value of the emotion parameter is set to the maximum value of emotion map 300 even once in a day. Also, state parameter acquisition section 112 adds 1 to DXM if the X value of the emotion parameter is set to the minimum value of emotion map 300 even once in a day, and adds 1 to DYM if the Y value of the emotion parameter is set to the minimum value of emotion map 300 even once in a day.

[0055] In this way, state parameter acquisition section 112 changes the amount of emotion change in accordance with a condition based on whether the emotion parameter value has reached the maximum or minimum value of emotion map 300. This update process changes the amount of emotion change, i.e., the degree of change in emotion.

[0056] For example, when only the head 204 is stroked repeatedly, only the emotion change amount DXP increases, and the other emotion change amounts do not change, so that the robot 200 has a personality that is easy to feel at ease. When only the head 204 is hit repeatedly, only the emotion change amount DXM increases, and the other emotion change amounts do not change, so that the robot 200 has a personality that is easy to feel anxious. In this way, the state parameter acquisition unit 112 changes the emotion change amount in response to various external stimuli.

[0057] (2) Personality parameters The personality parameters are parameters that represent the pseudo personality of the robot 200. The personality parameters include a plurality of personality values ​​each representing a different degree of personality. The state parameter acquisition unit 112 changes the plurality of personality values ​​included in the personality parameters in response to an external stimulus detected by the sensor unit 210.

[0058] Specifically, the state parameter acquisition unit 112 calculates the four personality values ​​according to the following (Equation 1). That is, the personality value (cheerful) is calculated by subtracting 10 from DXP, which indicates the tendency to feel at ease, the personality value (shy) is calculated by subtracting 10 from DXM, which indicates the tendency to become anxious, the personality value (active) is calculated by subtracting 10 from DYP, which indicates the tendency to become excited, and the personality value (spoiled) is calculated by subtracting 10 from DYM, which indicates the tendency to become lethargic. Personality score (cheerful) = DXP-10 Personality score (shy) = DXM-10 Personality score (active) = DYP-10 Personality score (spoiled) = DYM-10 …(Formula 1)

[0059] 6, a personality value radar chart 400 can be generated by plotting the personality value (cheerful) on the first axis, the personality value (active) on the second axis, the personality value (shy) on the third axis, and the personality value (spoiled) on the fourth axis. Since the initial value of each variable of the amount of emotion change is 10 and increases up to a maximum of 20, the range of the personality value is 0 to 10.

[0060] The personality value radar chart 400 is similar to that disclosed in JP 2023-115370, and description thereof will be omitted.

[0061] In this way, the robot 200 has various personalities depending on how the user interacts with the robot 200. In other words, the personalities of the robots 200 are formed differently depending on how the user interacts with them.

[0062] These four personality values ​​are fixed when the child period has passed and the simulated growth of the robot 200 is completed. In order to correct the personality in the subsequent adult period according to the user's attitude toward the robot 200, the state parameter acquisition unit 112 adjusts the four personality correction values ​​(cheerful correction value, active correction value, shy correction value, and spoiled correction value).

[0063] State parameter acquisition section 112 adjusts the four personality correction values ​​in accordance with conditions based on which area on emotion map 300 the emotion parameter has existed for the longest time. Specifically, the four personality correction values ​​are adjusted as follows (A) to (E). (A) If the longest existence area is a safe area on the emotion map 300, the state parameter acquisition section 112 adds 1 to the cheerful correction value and subtracts 1 from the shy correction value. (B) If the longest existence area is an excited area on the emotion map 300, the state parameter acquisition section 112 adds 1 to the active correction value and subtracts 1 from the spoiled correction value. (C) If the longest presence area is an anxious area on the emotion map 300, the state parameter acquisition section 112 adds 1 to the shy correction value and subtracts 1 from the cheerful correction value. (D) If the longest existence area is a lethargic area on the emotion map 300, the state parameter acquisition section 112 adds 1 to the spoiled correction value and subtracts 1 from the active correction value. (E) If the longest presence area is the central area on the emotion map 300, the state parameter acquisition section 112 decreases the absolute values ​​of all of the four personality correction values ​​by one.

[0064] When the four personality correction values ​​are set, the state parameter acquisition unit 112 calculates the four personality values ​​according to the following (Equation 2). Personality value (cheerful) = DXP-10 + cheerfulness correction value Personality score (shy) = DXM-10 + shyness correction value Personality score (active) = DYP-10 + Activeness correction value Personality score (spoiled) = DYM-10 + spoiled correction value …(Formula 2)

[0065] (3) Battery level The remaining battery level is the remaining amount of power stored in the battery 250, and is a parameter that indicates the pseudo hunger level of the robot 200. The state parameter acquisition unit 112 acquires information on the current remaining battery level from a power supply control unit that controls charging and discharging of the battery 250.

[0066] (4) Current location The current location is the location where the robot 200 is currently located. The state parameter acquisition unit 112 acquires information on the current location of the robot 200 by the position information acquisition unit 260.

[0067] When the current location matches the location most frequently recorded, the state parameter acquisition unit 112 determines that the current location is the home. When the current location is not the home, the state parameter acquisition unit 112 determines whether the current location is a new location, a familiar location, or a less familiar location based on the number of times the location has been recorded in the past, and acquires the determination information. For example, the state parameter acquisition unit 112 determines that the current location is a familiar location if the number of times the location has been recorded in the past is five or more, and determines that the current location is a less familiar location if the number of times the location has been recorded in the past is less than five.

[0068] (5)Current time The current time is the current time. The state parameter acquisition unit 112 acquires the current time from a clock mounted on the robot 200. Note that, like the acquisition of position information, the acquisition of the current time is not limited to this method.

[0069] More specifically, the state parameter acquisition unit 112 refers to the log of ON / OFF of the simulated sleep of the robot 200 to determine whether the current time is immediately after the wake-up time for this day, immediately before bedtime, or during a nap time.

[0070] (6) Number of days to grow (number of days to grow) The number of days of growth indicates the number of days of the pseudo growth of the robot 200. The robot 200 is pseudo-born when the user starts it for the first time after being shipped from the factory, and grows from a child to an adult over a predetermined growth period. The number of days of growth corresponds to the number of days from the pseudo birth of the robot 200.

[0071] The initial value of the number of days of growth is 1, and the state parameter acquisition unit 112 adds 1 to the number of days of growth every time a day passes. The growth period during which the robot 200 grows from a child to an adult is, for example, 50 days, and the period of 50 days of pseudo-growth from birth is called the "child period." When the child period has elapsed, the pseudo-growth of the robot 200 is completed. The period after the child period is completed is called the "adult period."

[0072] During the child period, the state parameter acquisition unit 112 increases both the maximum and minimum values ​​of the emotion map 300 by 2 each time the number of days of simulated growth of the robot 200 increases by one day. The initial size of the emotion map 300 has a maximum value of 100 and a minimum value of -100 for both the X and Y values, as shown in a frame 301. When the number of days of growth reaches half the child period (for example, 25 days), the maximum value of both the X and Y values ​​becomes 150 and the minimum value becomes -150, as shown in a frame 302. When the child period has elapsed, the simulated growth of the robot 200 stops. At this time, the maximum value of both the X and Y values ​​becomes 200 and the minimum value becomes -200, as shown in a frame 303. After that, the size of the emotion map 300 is fixed.

[0073] The settable range of emotion parameters is determined by emotion map 300. Therefore, the settable range of emotion parameters expands as the size of emotion map 300 increases. By expanding the settable range of emotion parameters, richer emotional expression becomes possible, and thus the pseudo growth of robot 200 is expressed by expanding the size of emotion map 300.

[0074] Returning to FIG. 3, the gesture control unit 113 causes the robot 200 to perform various gestures according to the situation based on the gesture information 121. The gesture information 121 is information that determines the gestures to be performed by the robot 200. Here, the gestures are the behavior, action, etc. of the robot 200. Specifically, the gestures are, as shown in FIG. 7, "head down", "peep", "head shake", "surprise", "happiness", "sadness", etc. In addition to the gestures shown in FIG. 7, various other gestures can be mentioned, such as "laughing", "anger", "sneezing", "breathing", etc. Each gesture is composed of a combination of multiple elements, which are movements or voice outputs.

[0075] The movement means a physical movement (motion) of the robot 200 executed by the driving unit 220. Specifically, the movement corresponds to moving the head 204 relative to the body 206 by the twist motor 221 or the up-down motor 222. The sound output means outputting various sounds such as cries from the speaker 231 of the output unit 230.

[0076] 7, the gesture information 121 defines gesture control parameters for each of a plurality of gestures that the robot 200 can perform. The gesture control parameters are parameters for causing the robot 200 to perform each gesture. The gesture control parameters define, for each element constituting a gesture, a movement parameter or a cry parameter, and a time (in milliseconds) for performing the element. The movement parameters define the movement angle of the twist motor 221 and the movement angle of the up-down motor 222. The cry parameters define the sound and volume.

[0077] As an example, when the robot 200 is made to perform the gesture of "bowing its head", the gesture control unit 113 first controls the twist motor 221 and the up-down motor 222 so that their angles become 0 after 100 milliseconds, and controls the up-down motor 222 so that its angle becomes -45 after 100 milliseconds. When the robot 200 is made to perform the gesture of "peep", the gesture control unit 113 outputs a "peep" sound at a volume of 60 dB from the speaker 231 for 300 milliseconds. When the robot 200 is made to perform the gesture of "shaking its head", the gesture control unit 113 first controls the twist motor 221 and the up-down motor 222 so that their angles become 0 after 100 milliseconds, and controls the drive unit 220 so that the angle of the twist motor 221 becomes 34 after 100 milliseconds, and controls the drive unit 220 so that the angle of the twist motor 221 becomes -34 after another 100 milliseconds.

[0078] 7 defines gestures such as "surprised," "happy," and "sad" as more complicated gestures. When making the robot 200 execute the gesture of "surprised," the gesture control unit 113 first controls the twist motor 221 and the up-down motor 222 so that their angles become 0 after 100 milliseconds, and controls the up-down motor 222 so that its angle becomes -24 after 100 milliseconds. The gesture control unit 113 then does not rotate for 700 milliseconds, and controls the twist motor 221 so that its angle becomes 34 and the up-down motor 222 so that its angle becomes -24 after 500 milliseconds. The gesture control unit 113 then controls the twist motor 221 so that its angle becomes -34 after 400 milliseconds, and controls the twist motor 221 and the up-down motor 222 so that their angles become 0 after 500 milliseconds. Furthermore, in parallel with driving the twist motor 221 and the up / down motor 222, the gesture control unit 113 outputs a sound "kyaa" at a volume of 70 dB from the speaker 231. Note that gesture control parameters for gestures such as "happy" and "sad" are omitted in Fig. 7, but like "surprised", these are determined by a combination of the movement (motion) by the twist motor 221 or the up / down motor 222 and the sound output (cry) from the speaker 231.

[0079] The gesture information 121 defines the gesture to be performed by the robot 200 by a combination of such actions (motions) or voice outputs (cries). The gesture information 121 may be pre-installed in the robot 200. Alternatively, the gesture information 121 may be freely created by the user operating the terminal device 50.

[0080] Each gesture defined in the gesture information 121 is associated in advance with a trigger, which is a condition for the robot 200 to execute the gesture. The trigger can be various conditions, specifically, "being called out to," "being stroked," "being lifted up," "being turned upside down," "it has become bright," "it has become dark," and the like. These triggers are triggers based on external stimuli and are detected by the sensor unit 210. For example, "being called out to" is detected by the microphone 213. "being stroked" is detected by the touch sensor 211. "being lifted up" and "being turned upside down" are detected by the acceleration sensor 212 or the gyro sensor 215. "it has become bright" and "it has become dark" are detected by the illuminance sensor 214. Note that the trigger may be one that is not based on an external stimulus, such as "a specific time has arrived," "the robot 200 has moved to a specific place," and the like.

[0081] The gesture control unit 113 judges whether or not any of the triggers of the multiple gestures defined in the gesture information 121 has been established based on the detection result by the sensor unit 210, etc. For example, the gesture control unit 113 judges whether or not any of the triggers defined in advance in the gesture information 121 has been established, such as whether the user's voice has been recognized, whether the head 204 of the robot 200 has been stroked, whether a specific time has arrived, whether the robot 200 has moved to a specific location, etc. As a result of the judgment, if any of the triggers has been established, the gesture control unit 113 causes the robot 200 to perform a gesture corresponding to the established trigger.

[0082] When any of the triggers is established, the gesture control unit 113 refers to the gesture information 121 to identify gesture control parameters set for the gesture corresponding to the established trigger. Specifically, the gesture control unit 113 identifies, as the gesture control parameters, a combination of actions or cries that are elements constituting the gesture corresponding to the established trigger, the execution start timing of each element, and action parameters or cries parameters that are parameters of each element. Then, the gesture control unit 113 drives the driving unit 220 based on the identified gesture control parameters, or outputs sound from the speaker 231, thereby causing the robot 200 to execute the gesture corresponding to the established trigger.

[0083] More specifically, the gesture control unit 113 corrects the gesture control parameters identified from the gesture information 121 based on the state parameters 122 acquired by the state parameter acquisition unit 112. This makes it possible to change the gestures according to the current state of the robot 200, and to realistically imitate living creatures.

[0084] In order to correct the gesture control parameters, the gesture control unit 113 refers to the coefficient table 124. As shown in Fig. 8 and Fig. 9, the coefficient table 124 defines a correction coefficient for each of the state parameters 122, that is, (1) emotion parameters, (2) personality parameters, (3) remaining battery level, (4) current location, and (5) current time. Although not shown, the coefficient table 124 may also define a correction coefficient for (6) number of days of growth.

[0085] The correction coefficient is a coefficient for correcting the gesture control parameter identified from the gesture information 121. Specifically, the correction coefficient is determined by the action direction and weighting coefficient for each of the speed and amplitude of the up-down movement by the up-down motor 222, the speed and amplitude of the left-right movement by the twist motor 221, and the movement start time lag.

[0086] More specifically, the gesture control unit 113 determines which of the following (1) to (5) corresponds to the current state of the robot 200 indicated by the state parameters 122 acquired by the state parameter acquisition unit 112. Then, the gesture control unit 113 corrects the gesture control parameters by using a correction coefficient corresponding to the current state of the robot 200. (1) Which of the following emotional parameters of the robot 200 corresponds to: happiness, frustration, sadness, lethargy, or normal. In other words, whether the coordinates (X, Y) representing the emotional parameters are located in the areas marked "happiness," "irritation," "sadness," "lethargy," or "normal" on the emotional map 300 shown in FIG. 5. (2) Which of the following personality parameters does the robot 200 currently have: cheerful, lively, shy, or clingy? In other words, which of the four personality values, cheerful, lively, shy, or clingy, is the largest? (3) Whether the current battery remaining capacity of the robot 200 is 70% or more, between 70% and 30%, or 30% or less. (4) Whether the robot 200 is currently located at home, in a familiar location, in an unfamiliar location, or in a new location. (5) Whether the current time is just after waking up, during a nap, or just before going to bed.

[0087] 9, when the current time corresponds to the time immediately after waking up, the action direction for both the speed and amplitude of the up-down movement and the left-right movement is set to "-" and the weighting coefficient is set to "0.2." Therefore, the gesture control unit 113 lengthens the movement time by 20% and shortens the movement distance by 20% based on the value acquired from the gesture information 121. In other words, the gesture control unit 113 makes the movement of the robot 200 20% slower and 20% shorter than usual.

[0088] 9, the direction of action of the action start time lag is set to "+" and the weighting coefficient is set to "0.2". Therefore, the gesture control unit 113 delays the execution start timing by 20% from the normal timing based on the value set in the gesture information 121. By performing correction using such a correction coefficient, it is possible to express that the gesture is executed slightly slower than the normal motion when the user is in a sleepy state immediately after waking up.

[0089] The gesture control unit 113 specifies the correction coefficients for the corresponding states of (1) emotion parameters, (2) personality parameters, (3) remaining battery level, and (4) current location in addition to (5) the current time from the coefficient table 124. Then, the gesture control unit 113 corrects the gesture control parameters using the sum of the corresponding correction coefficients for (1) to (5).

[0090] As a specific example, a case will be described where (1) the current emotion parameter corresponds to joy, (2) the current personality parameter corresponds to cheerfulness, (3) the current battery level corresponds to 30% or less, (4) the current location corresponds to a new location, and (5) the current time corresponds to immediately after waking up.

[0091] In this case, referring to the coefficient table 124 shown in Fig. 8 and Fig. 9, the sum of the correction coefficients for the speed and amplitude of the up-down movement is calculated as "+0.2+0.1-0.3-0.2-0.2=-0.4", and the sum of the correction coefficients for the speed and amplitude of the left-right movement is calculated as "+0.2+0-0.3-0.2-0.2=-0.5". Therefore, the gesture control unit 113 lengthens the operation time of the up-down motor 222 by 40% and shortens the operation distance by 40% based on the value set in the gesture information 121. Furthermore, the gesture control unit 113 lengthens the operation time of the twist motor 221 by 50% and shortens the operation distance by 50% based on the value acquired from the gesture information 121.

[0092] The sum of the correction coefficients for the action start time lag is calculated as +0+0+0.3+0.2+0.2=+0.7. Therefore, the gesture control unit 113 delays the execution start timing by 70% from the normal timing, using the value acquired from the gesture information 121 as a reference.

[0093] Although not shown, the coefficient table 124 also defines correction coefficients for cries in the same manner as for actions. To be more specific, the gesture control unit 113 corrects the volume, which is a cries parameter set for a gesture corresponding to the established trigger in the gesture information 121, by using a correction coefficient corresponding to the state parameter 122 acquired by the state parameter acquisition unit 112.

[0094] In this way, the gesture control unit 113 corrects the gesture control parameter based on the state parameter 122 acquired by the state parameter acquisition unit 112. Then, the gesture control unit 113 drives the driving unit 220 based on the corrected gesture control parameter, or outputs a sound from the speaker 231, thereby making the robot 200 execute a gesture corresponding to the established trigger. By correcting the gesture control parameter based on the state parameter 122, even when the robot 200 executes the same gesture, the gesture executed differs depending on the current state (emotion, personality, remaining battery level, current location, current time, etc.) of the robot 200. For example, even when the robot 200 executes the same "happy" gesture, the gesture executed differs between when the pseudo-emotion corresponds to "happy" and when the pseudo-emotion corresponds to "irritation". This prevents the gestures from becoming uniform, and allows individuality to be expressed.

[0095] Next, the link mode will be described. In the link mode, the robot 200, which performs various gestures as described above, performs gestures linked to images displayed on the terminal device 50. In the link mode, the robot system 1 is an example of a link system.

[0096] As an example of the cooperation mode, a case will be described below in which an image of a walk is displayed on the terminal device 50 and the robot 200 performs the gesture of taking a walk. However, the cooperation mode may be any mode as long as it can be executed in cooperation between the terminal device 50 and the robot 200. For example, an image of food being served may be displayed on the terminal device 50 and the robot 200 may perform the gesture of eating the food. Alternatively, an image of a PC may be displayed on the terminal device 50 and the robot 200 may perform the gesture of working on the PC. Also, an image of a room layout may be displayed on the terminal device 50 and the robot 200 may perform the gesture of moving around according to the room layout.

[0097] When executing the cooperative mode, the user places the robot 200, for example, on the terminal device 50 or in a position near the terminal device 50 so that the user can simultaneously view the display unit 540 of the terminal device 50 and the robot 200. In this state, when the user operates the operation unit 530 of the terminal device 50 to input an instruction to start the cooperative mode, the cooperative mode starts.

[0098] When the cooperative mode is started, the user operates operation unit 530 to select a desired content from among a plurality of contents contained in content data 521 stored in storage unit 520. Content data 521 has content executable in the cooperative mode. Specifically, as shown in FIG. 10, content data 521 defines a plurality of contents. Each of the plurality of contents has video data 523 and control data 525.

[0099] The video data 523 is data showing a video to be played on the terminal device 50. The control data 525 is data defining a gesture to be executed by the robot 200 in conjunction with the corresponding video data 523. One video data 523 and one control data 525 are associated as a set and constitute one content.

[0100] When a content is selected by the user, the interlocking control unit 513 reads out the video data 523 of the selected content, and displays the video indicated in the read out video data 523 on the display unit 540. In this way, the interlocking control unit 513 plays back the video of the selected content.

[0101] The interlocking control unit 513 causes the robot 200 to perform a gesture corresponding to the video (image) when the video (image) of the content selected in this manner is displayed on the display unit 540. As a result, the interlocking control unit 513 causes the robot 200 to perform a gesture corresponding to the video in conjunction with the playback of the video. To this end, the interlocking control unit 513 refers to the control data 525 in the selected content.

[0102] As shown in Fig. 11, the control data 525 defines an execution timing, a gesture (movement or cry), a playback speed coefficient, and a coefficient effective time in association with each other. In the control data 525, the execution timing indicates the timing at which the robot 200 executes a gesture associated with each execution timing during playback of a video, as the elapsed time from the start timing of the video. The gesture indicates a movement or cry that is a gesture executed by the robot 200 during playback of the video. The playback speed coefficient is a coefficient used when changing the playback speed of the video in accordance with the state parameters 122 of the robot 200. The coefficient effective time indicates the effective time of the playback speed coefficient.

[0103] When a video is displayed on the display unit 540, and one of the execution timings defined in the control data 525 arrives, the interlocking control unit 513 causes the robot 200 to perform a gesture associated with the execution timing. As a result, the interlocking control unit 513 causes the robot 200 to perform a gesture corresponding to the content of the video at a predetermined timing during playback of the video.

[0104] Specifically, when any of the execution timings arrives, the interlocking control unit 513 generates an execution command indicating a gesture linked to the execution timing and transmits the command to the control device 100 of the robot 200 via the communication unit 550. When the gesture control unit 113 of the robot 200 receives the execution command transmitted from the terminal device 50, the gesture control unit 113 causes the robot 200 to execute the gesture indicated in the received execution command. As a result, the robot 200 sequentially executes a plurality of gestures defined in the control data 525 at the corresponding execution timings defined in the control data 525. The execution command is a command for controlling the robot 200 to cause the robot 200 to execute a gesture corresponding to an image displayed on the display unit 540, and is an example of robot control information.

[0105] In the example of the control data 525 shown in FIG. 11, first, the interlocking control unit 513 makes the robot 200 execute the first gesture when the execution timing corresponding to the timing when the video starts is 0 seconds later. For example, when a video of a walk starts, the interlocking control unit 513 makes the robot 200 execute the gesture of starting a walk. In this way, the interlocking control unit 513 synchronizes the timing when the video starts and the timing when the robot 200 starts to execute the gesture corresponding to the video. In other words, the interlocking control unit 513 transmits an execution command to the robot 200 via the communication unit 550, which controls the timing when the video starts and the timing when the robot 200 starts to execute the gesture corresponding to the video to be simultaneous.

[0106] Thereafter, at each execution timing determined in the control data 525, the linkage control unit 513 causes the robot 200 to perform a gesture corresponding to the content of the video at that execution timing. For example, the linkage control unit 513 causes the robot 200 to perform a gesture of surprise at a timing when a scene in which the robot suddenly encounters an insect during a walk is played in the video.

[0107] Furthermore, the interlocking control unit 513 determines whether or not it is necessary to change the playback speed of the video at each execution timing by referring to the playback speed coefficient defined in the control data 525. For example, the interlocking control unit 513 does not change the playback speed when the playback speed coefficient linked to each execution timing is 1.0, changes the playback speed to 0.5 times faster when the playback speed coefficient is 0.5, changes the playback speed to 1.3 times faster when the playback speed coefficient is 1.3, and stops the playback of the video when the playback speed coefficient is 0. Note that when the interlocking control unit 513 changes the playback speed, it returns the playback speed to the original playback speed (playback speed coefficient=1.0) after the coefficient effective time defined in the control data 525 has elapsed.

[0108] In the robot 200, when the gesture control unit 113 makes the robot 200 execute a gesture corresponding to the video being played, similarly to the case other than the above-mentioned linked mode, the gesture control unit 113 acquires a gesture control parameter corresponding to the execution command received from the terminal device 50 by referring to the gesture information 121. Then, the gesture control unit 113 corrects the gesture control parameter based on the correction coefficient defined in the coefficient table 124 shown in FIG. 8 and FIG. 9, and drives the driving unit 220 or outputs a sound from the speaker 231 according to the corrected gesture control parameter. As a result, the gesture control unit 113 makes the robot 200 execute a gesture corresponding to the video being played by the corrected gesture control parameter. The details of the correction using the correction coefficient are the same as those in the case other than the above-mentioned linked mode. By correcting the gesture control parameter based on the state parameter 122 in this way, even when the robot 200 executes the same gesture, the gesture to be executed varies depending on the current state (emotion, personality, remaining battery level, current location, current time, etc.) of the robot 200.

[0109] In this case, if the gesture execution time changes significantly due to the correction of the gesture control parameter, there is a possibility that synchronization will not be achieved between the playback of the video and the gesture of the robot 200. To avoid this, the gesture control unit 113 notifies the terminal device 50 of the change in execution time caused by the correction of the gesture control parameter. In the terminal device 50, upon receiving the notification from the robot 200, the interlocking control unit 513 corrects the coefficient valid time in accordance with the change in execution time.

[0110] As an example, when an insect appears during playback of a video of a walk and the robot 200 performs a gesture of being startled and stopping, the interlocking control unit 513 stops the progress of the walk in the video. In this case, if the time for stopping in surprise is usually 3 seconds, but the time for stopping is extended to 5 seconds due to correction of the gesture control parameter, the interlocking control unit 513 extends the scene in the video where the progress of the walk is stopped by 2 seconds, and delays the playback timing of all subsequent videos by 2 seconds. This makes it possible to prevent the synchronization between the playback of the video and the gestures of the robot 200 from being lost.

[0111] Next, the flow of the robot control process will be described with reference to Fig. 12. The robot control process shown in Fig. 12 is executed by the control unit 110 of the control device 100 when the user turns on the power of the robot 200. The robot control process is an example of a method for controlling a robot.

[0112] When the robot control process is started, the control unit 110 sets the state parameters 122 (step S101). When the robot 200 is started for the first time (when the robot is started for the first time by the user after being shipped from the factory), the control unit 110 sets each of the emotion parameters, personality parameters, and number of days to growth to an initial value (for example, 0). On the other hand, when the robot is started for the second time or later, the control unit 110 reads out the values ​​of each parameter saved in step S106 of the previous robot control process (described later) and sets them as the state parameters 122. However, the emotion parameters may all be initialized to 0 every time the power is turned on.

[0113] After setting the state parameters 122, the control unit 110 communicates with the terminal device 50 and acquires the gesture information 121 created on the basis of a user's operation in the terminal device 50 (step S102). Note that if the gesture information 121 has already been saved in the storage unit 120, step S102 may be skipped.

[0114] When the control unit 110 acquires the gesture information 121, the control unit 110 determines whether or not any one of the multiple gesture triggers defined in the gesture information 121 has been established (step S103).

[0115] When any of the triggers is established (step S103; YES), the control unit 110 executes a gesture control process and causes the robot 200 to execute a gesture corresponding to the established trigger (step S104). Details of the gesture control process in step S104 will be described with reference to the flowchart of Fig. 13. Step S104 is an example of a control step.

[0116] When the gesture control process shown in FIG. 13 is started, control unit 110 updates state parameters 122 (step S201). Specifically, if the trigger established in step S103 is due to an external stimulus, control unit 110 derives an emotion change amount corresponding to the external stimulus. Then, control unit 110 updates the emotion parameter by adding or subtracting the derived emotion change amount to or from the current emotion parameter. Furthermore, during the child period, control unit 110 calculates each personality value of the personality parameter from the emotion change amount updated in step S108 according to the above-mentioned (Equation 1). On the other hand, during the adult period, control unit 110 calculates each personality value of the personality parameter from the emotion change amount updated in step S108 and the personality correction value according to the above-mentioned (Equation 2).

[0117] When the state parameters 122 are updated, the control unit 110 refers to the gesture information 121 to acquire gesture control parameters of the gesture corresponding to the established trigger (step S202). Specifically, the control unit 110 acquires from the gesture information 121 a combination of actions or cries that are elements constituting the gesture corresponding to the established trigger, the execution start timing of each element, and the action parameters or cries parameters.

[0118] When the gesture control parameters are acquired, the control unit 110 corrects the gesture control parameters based on the correction coefficients defined in the coefficient table 124 (step S203). Specifically, the control unit 110 calculates the sum of the correction coefficients corresponding to the state parameters 122 updated in step S201, among the correction coefficients defined in the coefficient table 124 for (1) emotion parameters, (2) personality parameters, (3) remaining battery level, (4) current location, and (5) current time. Then, the control unit 110 corrects the movement parameters, the cry parameters, and the execution start timing based on the calculated sum of the correction coefficients.

[0119] After correcting the gesture control parameter, the control unit 110 executes a gesture corresponding to the established trigger (step S204). Specifically, the control unit 110 drives the driving unit 220 or outputs a sound from the speaker 231 according to the gesture control parameter corrected in step S203. This ends the gesture control process shown in FIG.

[0120] Returning to FIG. 12, in step S103, if none of the multiple gesture triggers is established (step S103; NO), control unit 110 skips step S104.

[0121] Next, the control unit 110 determines whether or not to end the process (step S105). For example, when the operation unit 240 receives an instruction from the user to turn off the power of the robot 200, the process ends. When the process ends (step S105; YES), the control unit 110 stores the current state parameters 122 in the non-volatile memory of the storage unit 120 (step S106) and ends the robot control process shown in FIG.

[0122] If the process does not end (step S105; NO), the control unit 110 determines whether the date has changed using the clock function (step S109). If the date has not changed (step S109; NO), the process returns to step S103.

[0123] If the date has changed (step S109; YES), the control unit 110 updates the state parameters 122 (step S110). To be more specific, if it is during the child period (e.g., 50 days after birth), the control unit 110 changes the values ​​of the emotion changes DXP, DXM, DYP, and DYM depending on whether the emotion parameters have reached the maximum or minimum values ​​of the emotion map 300. Also, if it is during the child period, the control unit 110 expands both the maximum and minimum values ​​of the emotion map 300 by a predetermined increment (e.g., 2). On the other hand, if it is during the adult period, the control unit 110 adjusts the personality correction value.

[0124] After updating the state parameter 122, the control unit 110 adds 1 to the number of days of growth (step S111) and returns to step S103. Then, the control unit 110 repeats the processes from step S103 to step S111 as long as the robot 200 is operating normally.

[0125] 14, a flow of the interlocking control process executed by the terminal device 50 and the robot 200 will be described. The interlocking control process starts when the user operates the terminal device 50 to input an instruction to start the interlocking mode.

[0126] When the interlocking control process is started, in terminal device 50, control unit 510 accepts, based on a user's operation, a selection of content to be executed in the interlocking mode from among a plurality of contents defined in content data 521 (step S301).

[0127] When the selection of the content is accepted, the control unit 510 communicates with the robot 200 via the communication unit 550 and notifies the robot 200 of the start of the linked mode (step S302). Then, the control unit 510 reads out the video data 523 of the selected content and starts playing the video (step S303).

[0128] When playback of the video starts, the control unit 510 refers to the control data 525 of the selected content and determines whether or not the timing for the robot 200 to perform a gesture has arrived (step S304).

[0129] When the timing to perform a gesture has arrived (step S304; YES), the control unit 510 transmits to the robot 200 an execution command indicating a gesture to be performed by the robot 200 (step S305).

[0130] In the control device 100 of the robot 200, when the control unit 110 receives an execution command transmitted from the terminal device 50, the control unit 110 causes the robot 200 to execute a gesture indicated in the received execution command (step S401). In step S401, the control unit 110 executes the gesture control process (steps S201 to S205) shown in FIG. 12, similarly to step S104.

[0131] In the terminal device 50, when the control unit 510 transmits the execution command, the control unit 510 determines whether or not the playback speed of the video needs to be changed (step S306). Specifically, the control unit 510 refers to the control data 525, and determines that the playback speed of the video needs to be changed when the playback speed coefficient associated with the arriving execution timing is a value other than 1.0.

[0132] If the playback speed needs to be changed (step S306; YES), control unit 510 changes the playback speed of the video being played back (step S307) in accordance with a playback speed coefficient defined in control data 525. On the other hand, if the playback speed does not need to be changed (step S306; NO), control unit 510 skips step S307.

[0133] In step S304, if the timing to perform the gesture has not arrived (step S304; NO), control unit 510 skips steps S305-S307.

[0134] Next, control unit 510 determines whether to end playback of the video (step S308). Specifically, control unit 510 determines whether the last scene of the video being played has been reached. If playback of the video is not to be ended (step S308; NO), control unit 510 returns the process to step S304, and repeats the processes of steps S304 to S307 while the video is being played.

[0135] Finally, when the reproduction of the video is to be ended (step S308; YES), the control unit 510 notifies the robot 200 of the end of the cooperation mode (step S309), and ends the cooperation mode. With the above, the cooperation control process shown in FIG. 14 ends.

[0136] As described above, the terminal device 50 according to the first embodiment displays a video on the display unit 540, and causes the robot 200 to perform a gesture corresponding to the video in conjunction with the playback of the video. This enables the robot 200, which is an entity, to perform gestures that are linked to various worldviews created by the contents of the digital content, namely the video. As a result, the interest of the robot 200 can be improved.

[0137] In particular, as a conventional technology for linking the contents of digital content such as video with the movements of a robot or the like, it is known that such technology can be realized by AR (Augmented Reality) technology, VR (Virtual Reality) technology, etc. However, in the case of AR technology or VR technology, the user watches a robot displayed on a screen. In contrast, in the first embodiment, the robot 200, which is an actual entity, performs gestures in conjunction with the video displayed on the terminal device 50, not on the screen of the terminal device 50, so that the sense of reality, cuteness, healing effect, etc. obtained from the gestures of the robot 200 can be improved.

[0138] Furthermore, the robot 200 according to the first embodiment changes its behavior in response to the state parameters 122 such as simulated emotions and personality when linked to a video. This allows for a wide variety of behaviors that reflect individuality rather than uniformity and that change according to various environmental conditions, thereby realizing long-term utility value that is unlikely to become boring.

[0139] (Embodiment 2) Next, a description will be given of embodiment 2. Descriptions of configurations and functions similar to those of embodiment 1 will be omitted where appropriate.

[0140] In the above-mentioned embodiment 1, the interlocking control unit 513 causes the robot 200 to perform a gesture in conjunction with the playback of a video displayed on the terminal device 50. In place of or in addition to such a feature, in embodiment 2, when an image is displayed on the display unit 540 and the interlocking control unit 513 detects that an operation related to the image has been received from the user by the operation unit 530, the interlocking control unit 513 controls the gesture of the robot 200 based on the image and the received operation.

[0141] Each of the multiple contents defined in the content data 521 according to the second embodiment has image data 524 and control data 525, as shown in Fig. 15. The image data 524 is data representing an image to be displayed on the terminal device 50.

[0142] In the first embodiment, each content has video data 523. In contrast, in the second embodiment, each content has image data 524. To be more specific, the image displayed on the terminal device 50 in the second embodiment is not limited to a video, and may be a still image. In the following, as an example, a case will be described in which the display unit 540 displays a still image as an image, and the displayed still image is switched in response to a user operation.

[0143] In the second embodiment, similarly to the first embodiment, the user selects a content to be executed in the linked mode from among a plurality of contents included in the content data 521. When an image of the selected content is displayed on the display unit 540 of the terminal device 50, the terminal device 50 accepts an operation from the user.

[0144] As an example, a case will be described where an image related to food is displayed on the terminal device 50. Specifically, as shown in Fig. 16, the interlocking control unit 513 displays an image of food (corn in the example of Fig. 16) on the display unit 540 of the terminal device 50. When such an image of food is displayed on the terminal device 50, the interlocking control unit 513 causes the robot 200 to perform a gesture of eating the food as a gesture corresponding to the image of the food.

[0145] More specifically, when such an image is displayed, the user operates the operation unit 530 to input a user operation (hereinafter referred to as a "first operation") related to the displayed image. Specifically, as the first operation, the user touches the touch panel or clicks and drags the mouse to input an operation to display an object OB within the image displayed on the display unit 540. Here, the object OB is, as an example, food that the user wishes the robot 200 to eat.

[0146] When the interlocking control unit 513 receives a first operation while an image is displayed on the display unit 540, the interlocking control unit 513 refers to the control data 525 corresponding to the image data 524, updates the image displayed on the display unit 540, and further controls the robot 200. In the second embodiment, the control data 525 defines update information of the image and control information of the robot 200 when a first operation is received from a user while an image of the corresponding image data 524 is displayed. The control information of the control data 525 links the image displayed on the display unit 540, the first operation received from the user, and a gesture to be executed by the robot 200. When the interlocking control unit 513 is displaying an image on the display unit 540 and detects that the operation unit 530 has received a first operation, the interlocking control unit 513 refers to the control information of the control data 525 and transmits an execution command to the robot 200 via the communication unit 550 to cause the robot 200 to execute a gesture based on the image and the first operation.

[0147] When detecting that a first operation has been received by the operation unit 530, first, the interlocking control unit 513 updates the image displayed on the display unit 540 based on the first operation. For example, when a food object OB is displayed by the first operation, the interlocking control unit 513 displays the object OB on the display unit 540 as shown in Fig. 17. In the example of Fig. 17, the object OB is a piece of corn.

[0148] Secondly, when it is detected that the operation unit 530 has accepted the first operation, the interlocking control unit 513 controls the gesture of the robot 200 based on the first operation. Specifically, the interlocking control unit 513 generates an execution command for making the robot 200 execute a gesture corresponding to the accepted first operation, and transmits the execution command to the robot 200. For example, when a food object OB is displayed by the first operation, the interlocking control unit 513 makes the robot 200 execute a gesture of eating the displayed object OB.

[0149] In the control device 100 of the robot 200, when the gesture control unit 113 receives the execution command transmitted from the terminal device 50, the gesture control unit 113 causes the robot 200 to execute a gesture in accordance with the received execution command. Specifically, as shown in Fig. 18, the gesture control unit 113 moves the robot 200 to a position where the object OB is displayed on the display unit 540 of the terminal device 50, and causes the robot 200 to execute the gesture of eating the object OB.

[0150] After controlling the gesture of the robot 200 based on the first operation, the interlocking control unit 513 updates the object OB in the image. For example, when the robot 200 executes a gesture of eating the object OB, the interlocking control unit 513 removes the object OB from the image so that it is not displayed on the display unit 540, as shown in Fig. 19. In this way, the interlocking control unit 513 causes the image displayed on the display unit 540 to correspond to the robot 200 executing a gesture of eating the object OB.

[0151] As described above, when an operation related to an image displayed on the display unit 540 is received from a user, the terminal device 50 according to the second embodiment controls the behavior of the robot 200 based on the received operation. In this way, the user's operation related to the image displayed on the terminal device 50 and the behavior of the robot 200 can be linked, so that the interest of the robot 200 can be further improved.

[0152] (Embodiment 3) Next, a description will be given of embodiment 3. Descriptions of configurations and functions similar to those of embodiments 1 and 2 will be omitted where appropriate.

[0153] In the above-mentioned second embodiment, the interlocking control unit 513 controls the behavior of the robot 200 based on a user's operation related to an image accepted when a video is displayed on the display unit 540. In place of or in addition to such a feature, in the third embodiment, when an image is displayed on the display unit 540 and an external stimulus to the robot 200 is detected by the sensor unit 210, the interlocking control unit 513 updates the image displayed on the display unit 540 based on the image and the external stimulus.

[0154] In the third embodiment, similarly to the first and second embodiments, the user selects a content to be executed in the linked mode from among a plurality of contents included in the content data 521. When an image of the selected content is displayed on the display unit 540 of the terminal device 50, the robot 200 uses various sensors provided in the sensor unit 210 to accept an operation from the user.

[0155] As an example, a case will be described where an image related to a walk is displayed on the terminal device 50. Specifically, as shown in Fig. 20, a path, scenery, etc. that is a target for a walk are displayed on the display unit 540 of the terminal device 50. The robot 200 performs a gesture of moving along the path displayed on the display unit 540.

[0156] When a scene of a branching point where multiple paths branch out is displayed in the walk image, the user moves the orientation of the body 206 of the robot 200 in the direction of the path along which the robot 200 should proceed. The control unit 110 detects such a user's operation on the robot 200 (hereinafter referred to as a "second operation") as an external stimulus by the acceleration sensor 212 or the gyro sensor 215.

[0157] When the control unit 110 accepts the second operation, it transmits operation information indicating the content of the accepted second operation to the terminal device 50 via the communication unit 130. When the interlocking control unit 513 in the terminal device 50 receives the operation information transmitted from the robot 200, it updates the image displayed on the display unit 540 based on the second operation indicated in the received operation information.

[0158] Specifically, when operation information indicating the direction of the road along which the robot 200 should proceed at a branching point is received, the interlocking control unit 513 updates the image displayed on the display unit 540 from the image of the scene at the branching point to an image of a scene proceeding in the direction indicated by the operation information. As an example, as shown in Fig. 21, when the user inputs an operation to turn the robot 200 to the left as the second operation, the interlocking control unit 513 updates the image displayed on the display unit 540 to an image of a scene proceeding to the left from the branching point.

[0159] As described above, when the terminal device 50 according to the third embodiment receives a user's operation on the robot 200 while an image is displayed on the display unit 540, the terminal device 50 updates the image displayed on the display unit 540 based on the received operation. In this way, the user's operation on the robot 200 and the image displayed on the terminal device 50 can be linked, so that the interest of the robot 200 can be further improved.

[0160] (Modification) Although the embodiment of the present invention has been described above, the above embodiment is merely an example, and the scope of application of the present invention is not limited to this. In other words, the embodiment of the present invention can be applied in various ways, and all embodiments are included in the scope of the present invention.

[0161] For example, in the above embodiment, an example has been described in which one robot 200 cooperates with the terminal device 50 in the cooperation mode. However, a plurality of robots 200 may cooperate with the terminal device 50 by performing the same gesture or different gestures in cooperation with each other.

[0162] In the above embodiment, the control device 100 is built into the robot 200, but the control device 100 may be a separate device (e.g., a server) without being built into the robot 200. When the control device 100 is located outside the robot 200, the control device 100 communicates with the robot 200 via the communication unit 130 to send and receive data to and from the robot 200, and controls the robot 200 as described in the above embodiment.

[0163] In the above embodiment, the exterior 201 is formed in a cylindrical shape from the head 204 to the torso 206, and the robot 200 has a crawling shape. However, the robot 200 is not limited to imitating a crawling creature. For example, the robot 200 may have a shape with arms and legs and imitate a four-legged or two-legged creature.

[0164] Furthermore, the robot is not limited to the robot 200 that imitates a living thing. For example, the robot may be a robot that imitates a wristwatch, stationery, an imaging device such as a camera, a projector, etc., as long as the robot is capable of expressing its individuality by performing various actions. Even if the robot is other than the robot 200, it can be described in the same manner as the above embodiment by having the same configuration and function as the above-mentioned robot 200.

[0165] In the above embodiment, the control unit 110 functions as each unit such as the gesture control unit 113 by the CPU executing a program stored in the ROM. Also, the control unit 510 functions as the interlocking control unit 513 by the CPU executing a program stored in the ROM. However, in the present invention, the control units 110 and 510 may have dedicated hardware such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or various control circuits instead of a CPU, and the dedicated hardware may function as each unit such as the gesture control unit 113. In this case, the functions of each unit may be realized by individual hardware, or the functions of each unit may be realized by a single hardware. Also, some of the functions of each unit may be realized by dedicated hardware, and the other parts may be realized by software or firmware.

[0166] It is possible to provide the robot 200 or the terminal device 50 equipped with a configuration for realizing the functions according to the present invention, and also possible to make an existing information processing device or the like function as the robot 200 or the terminal device 50 according to the present invention by applying a program. That is, by applying a program for realizing each functional configuration of the robot 200 or the terminal device 50 exemplified in the above embodiment so that it can be executed by a CPU or the like that controls the existing information processing device or the like, it is possible to make it function as the robot 200 or the terminal device 50 according to the present invention.

[0167] Moreover, the method of applying such a program is arbitrary. For example, the program can be applied by storing it in a computer-readable storage medium such as a flexible disk, a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc)-ROM, or a memory card. Furthermore, the program can be superimposed on a carrier wave and applied via a communication medium such as the Internet. For example, the program can be posted and distributed on a bulletin board (BBS: Bulletin Board System) on a communication network. Then, the program can be started and executed under the control of an OS (Operating System) in the same way as other application programs, thereby enabling the above-mentioned processing to be executed.

[0168] The above describes preferred embodiments of the present invention. However, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the claims. [Explanation of symbols]

[0169] 1...robot system, 50...terminal device, 100...control device, 110...control unit, 112...state parameter acquisition unit, 113...gesture control unit, 120...storage unit, 121...gesture information, 122...state parameters, 124...coefficient table, 130...communication unit, 200...robot, 201...exterior, 202...decorative part, 203...fur, 204...head, 205...connection unit, 206...torso unit, 207...casing, 210...sensor unit, 211...touch sensor, 212...acceleration sensor, 213...microphone, 214...illumination sensor, 215...gyro sensor, 220...drive unit, 221...twist motor, 222...up and down motor, 230...output unit, 231... Speaker, 240...operation unit, 250...battery, 260...location information acquisition unit, 300...emotion map, 301-303...frames, 400...personality value radar chart, 510...control unit, 513...linkage control unit, 520...storage unit, 521...content data, 523...video data, 524...image data, 525...control data, 530...operation unit, 540...display unit, 550...communication unit, BL...bus line, OB...object

Claims

1. A display means for displaying an image; A transmitting means for transmitting information to the robot; a control means for transmitting, to the robot via the transmission means, robot control information for causing the robot to perform a gesture corresponding to the image when the image is displayed on the display means; A terminal device comprising:

2. the display means displays a moving image as the image; the control means transmits to the robot via the transmission means the robot control information for controlling the robot so that a timing at which the video is started and a timing at which the robot is caused to perform the gesture are simultaneous; 2. The terminal device according to claim 1 .

3. the display means displays a moving image as the image; the control means transmits, to the robot via the transmission means, the robot control information for controlling the robot to perform a gesture corresponding to the content of the video at a predetermined timing during playback of the video; 2. The terminal device according to claim 1 .

4. When the control means causes the display means to display the image and detects that a first operation, which is a user's operation related to the image, has been accepted by a predetermined accepting means, the control means transmits to the robot via the transmitting means the robot control information for causing the robot to execute the gesture based on the image and the first operation.

4. The terminal device according to claim 1, wherein the first and second terminals are connected to each other.

5. The control means when it is detected that the first operation has been accepted by the predetermined accepting means, an object is displayed within the image displayed on the display means; updating the object after causing the robot to perform the gesture based on the image and the first operation; 5. The terminal device according to claim 4.

6. when the control means is causing the display means to display the image and when a predetermined detection means detects an external stimulus to the robot, the control means updates the image displayed on the display means based on the image and the external stimulus.

4. The terminal device according to claim 1, wherein the first and second terminals are connected to each other.

7. A method for linking an image displayed on a terminal device with a robot, comprising the steps of: causing the terminal device to display the image; transmitting, to the robot, robot control information for causing the robot to perform a gesture corresponding to the image when the image is displayed on the terminal device; A linking method comprising:

8. Computer, A display means for displaying an image; A transmitting means for transmitting information to the robot; a control means for transmitting, to the robot via the transmission means, robot control information for causing the robot to perform a gesture corresponding to the image when the image is displayed on the display means; A program to function as a

Images