Avatar robot dialogue system, control program, and control method

Abstract

This system provides an avatar robot dialogue system that can effectively deter abusive acts against avatar robots. [Solution] An avatar robot (100) is placed in the dialogue area of ​​the dialogue system (10), and the user (USR) can interact with the operator (OP) through the avatar robot. A determination means (24, 50d, S3) determines whether the user or a third party is committing an act of interference or infringement against the avatar robot. For example, by collecting data such as the video signal from the avatar robot's camera (116), the audio signal from the microphone (120), and the pressure signal from the pressure sensor (140), the determination means determines whether or not an infringement has occurred based on this data, for example using a neural network. A deterrent action execution means (24, 50e, S5, 126, S27-29) causes the avatar robot to perform a deterrent action (such as voice or physical movement) to deter infringement.

Description

Technical Field

[0001] The present disclosure relates to an avatar robot interaction system, a control program, and a control method, and particularly, for example, to an avatar robot interaction system, a control program, and a control method in which an operator has a conversation with a user (person) through the avatar robot.

Background Art

[0002] As the realism of avatar robots increases, antisocial acts against the avatar robots, such as violent acts and indecent acts, are more likely to occur. These acts lead to problems such as destruction of the avatar robot, and discomfort experienced by surrounding people or remotely operating operators.

[0003] For example, Non-Patent Document 1 conducts a sociological study on bullying acts performed by children on autonomous avatar robots.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] It is necessary to prevent such antisocial acts (infringing acts) against the avatar robot.

[0006] Therefore, a main object of the present disclosure is to provide a novel avatar robot interaction system, control program, and control method.

[0007] Another object of this disclosure is to provide an avatar robot interaction system, control program, and control method that can deter infringement of avatar robots. [Means for solving the problem]

[0008] To address the above-mentioned issues, this disclosure employs the following configuration. Reference numerals and supplementary explanations in parentheses indicate correspondences with embodiments described to aid understanding of this disclosure and do not limit this disclosure in any way.

[0009] The first embodiment is an avatar robot dialogue system in which a user interacts with an avatar robot controlled by an operator terminal operated by an operator, and comprises a determination means for determining whether there is an infringing act against the avatar robot, and a deterrent action execution means for causing the avatar robot to perform a deterrent action to deter the infringing act when it is determined that there is an infringing act.

[0010] In the dialogue system of the first embodiment (10: a reference numeral used to illustrate a corresponding part in the embodiment, not intended to limit it; the same applies hereinafter), an avatar robot (100) is placed in the dialogue area, and a user (USR) can interact with an operator (OP) through the avatar robot. A determination means (24, 50d, S3) determines whether the user or a third party is committing an act of interference or infringement against the avatar robot. For example, by collecting data such as video signals from the avatar robot's camera (116), audio signals from the microphone (120), and pressure signals from the pressure sensor (140), the determination means determines the presence or absence of an infringement based on this data, for example, by using a neural network. A deterrent action execution means (24, 50e, S5, 126, S27-29) causes the avatar robot to perform a deterrent action (such as voice or physical movement) to deter an infringement.

[0011] According to the first embodiment, infringing acts against avatar robots can be effectively deterred.

[0012] The second embodiment is an avatar robot dialogue system dependent on the first embodiment, comprising at least one of a camera that photographs the area around the avatar robot, a microphone that acquires sound from the area around the avatar robot, and a pressure sensor that detects contact with the avatar robot, wherein the determination means determines whether there is an infringing act based on at least one of the video signal from the camera, the audio signal from the microphone, and the pressure signal from the pressure sensor.

[0013] In the second embodiment, cameras (18, 116) capture images of the area around the avatar robot (100) and output video signals, a microphone (120) acquires sound from the area around the avatar robot (100) and outputs audio signals, and a pressure sensor (140) detects contact with the avatar robot (100) and outputs a pressure signal. A determination means (24, 50d, S3) determines whether an infringing act has occurred based on at least one of these video signals, audio signals, and pressure signals.

[0014] According to the second embodiment, it is possible to effectively determine whether or not an infringing act has occurred against the avatar robot.

[0015] A third embodiment is an avatar robot dialogue system dependent on the first embodiment, wherein the deterrent action execution means includes at least one of a vocalization means that causes the robot to make a sound to deter abusive behavior and a body movement means that causes the robot to perform a body movement to deter abusive behavior.

[0016] In the third embodiment, the vocal means (118) can vocalize, for example, a stop, a request for help, a warning, advice, or even a direct or indirect reference to the judgment result of the infringing act, in order to deter the infringing act. The bodily movement means (138, 142-152) can, in order to deter the infringing act, make the avatar robot (100) make facial expressions such as anger, confusion, or sadness, if the intention is to deter the infringing act with the avatar robot (100). Alternatively, if the intention is to deter the act with the movements of the avatar robot (100), the avatar robot (100) can make movements such as covering its face with both hands and turning its face away, crossing its arms to convey anger, crossing its arms or lowering its head to defend itself, thrusting both hands towards the other party to stop them, making an "X" with both hands or fingers, or closing its eyes or stopping to indicate that it will not engage with the other party. However, infringement deterrent actions can be carried out by at least one of the following: vocalization using such vocal means and bodily movements using bodily movement means (including facial expressions).

[0017] According to the third embodiment, abusive acts against the avatar robot can be effectively deterred by vocalization means and / or bodily movement means.

[0018] The fourth embodiment is an avatar robot dialogue system dependent on the first embodiment, wherein the determination means includes a severity determination means for determining the severity of an infringing act, and the deterrent action execution means causes the deterrent action to be carried out according to the severity.

[0019] In the fourth embodiment, the severity determination means associates (pairs) the presented information on infringing acts and their severity, uses a classification tree such as a deep neural network to perform supervised learning or semi-supervised learning, models the relationship between the paired information, and stores it as reference data. When multimodal sensor data (information) is input, it can determine how close the event is to the infringing acts presented in advance, and estimate the severity thereof. The deterrent action execution means causes the avatar robot (100) to perform words and actions for deterring the infringing act in accordance with the determined category of the infringing act and the severity thereof, etc.

[0020] According to the fourth embodiment, it is possible to execute an appropriate deterrent action corresponding to the severity of the infringing act. [Advantageous Effects of Invention]

[0021] According to the present disclosure, infringing acts against the avatar robot can be effectively deterred.

[0022] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description of the embodiments with reference to the drawings. [Brief Description of Drawings]

[0023] [Figure 1] FIG. 1 is an illustrative diagram showing an example of an avatar robot interaction system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is an illustrative diagram showing an example of an installation state of the avatar robot in the avatar robot interaction system of the embodiment of FIG. 1. [Figure 3] FIG. 3 is an illustrative diagram showing an example of the avatar robot interacting with a user in the avatar robot interaction system of the embodiment of FIG. 1. [Figure 4] FIG. 4 is a block diagram showing an example of a configuration of an avatar robot controller that controls the avatar robot in the embodiment of FIG. 3. [Figure 5]Figure 5 is a block diagram showing an example of an operator terminal in the avatar robot dialogue system of the embodiment shown in Figure 1. [Figure 6] Figure 6 is an illustrative diagram showing an example of the memory map of the operator terminal shown in Figure 5. [Figure 7] Figure 7 is an illustrative diagram showing an example of the memory map of the avatar robot shown in Figure 4. [Figure 8] Figure 8 is a flowchart showing an example of the operation of the operator terminal shown in Figure 5. [Figure 9] Figure 9 is a flowchart showing an example of the operation of the avatar robot shown in Figure 4. [Modes for carrying out the invention]

[0024] Referring to Figure 1, in one embodiment of the present invention, the avatar robot dialogue system 10 communicates with the operator terminal 14 and the avatar robot 100 via a network 12 such as the Internet. For example, the operator OP interacts with the user USR (Figure 2) through the avatar robot 100 from the operator terminal 14.

[0025] As shown in Figure 2, the avatar robot 100 is, for example, placed on a table 16, and the user USR faces the avatar robot 100 and interacts with the operator OP.

[0026] The avatar robot 100 incorporates, for example, an avatar robot controller 122 shown in Figure 4, and controls the avatar robot 100 using the avatar robot controller 122 in response to commands from the operator terminal 14 shown in Figure 5.

[0027] In the dialogue area of ​​the embodiment, an environmental camera 18 is provided on the table 16, which can capture images of the space between the avatar robot 100 and the user USR. The environmental camera 18 is primarily provided to capture infringing acts against the avatar robot 100 by the user USR or a third party (not shown), such as touching, hitting, or shaking the avatar robot 100, or any other act that interferes with the avatar robot 100's operation.

[0028] An emergency switch 20 is also installed on the table 16. This emergency switch 20 is basically operated by a third party. For example, if a third party discovers an infringement against the avatar robot 100, they can operate this emergency switch 20 to notify the operator terminal 14 that an infringement is occurring against the avatar robot 100.

[0029] Here, with reference to Figures 3 and 4, the avatar robot 100 will be described to the extent necessary for understanding this embodiment.

[0030] However, it should be noted in advance that the avatar robot 100 shown in Figures 3 and 4 is merely an example, and other types of avatar robots 100 are also available. For example, ERICA (product name), CommU (product name), Sota (product name), etc.

[0031] Referring to Figure 3, the avatar robot 100 is mounted on the base 102 so that it can rotate forward, backward, left, and right relative to the base 102. In other words, the torso 104 has two degrees of freedom.

[0032] From the left and right positions of the torso 104 corresponding to the shoulders of a person, the right arm 106R and the left arm 106L are respectively provided by shoulder joints (not shown), allowing them to rotate forward, backward, left, and right. In other words, the right arm 106R and the left arm 106L each have two degrees of freedom.

[0033] A neck (not shown) is provided at the upper center of the torso 104, and a head 108 is provided above it. The neck, or head 108, is attached to the torso 104 so that it can rotate forward, backward, left, and right. In other words, the head 108 has three degrees of freedom: roll angle (tilting left and right), pitch angle (tilting forward and backward), and yaw (rotation left and right).

[0034] The front surface of the head 108, which corresponds to the human face, is provided with a right eye 110R and a left eye 110L, and the right eye 110R and left eye 110L are provided with eyeballs 112R and 112L. The right eye 110R and left eye 110L can open and close their eyelids, and the eyeballs 112R and 112L can rotate up, down, left, and right, respectively. In other words, the right eye 110R and left eye 110L, i.e., the eyelids, have 1 degree of freedom, and the eyeballs 112R and 112L have 2 degrees of freedom.

[0035] The face also has a mouth 114, which can be opened and closed. In other words, the mouth 114 has 1 degree of freedom.

[0036] A camera 116 is built into the front of the face of the head 108, and this camera 116 is capable of photographing people facing the avatar robot 100, as well as people and scenery behind them.

[0037] A speaker 118 is built into the mouth 114, and a microphone 120 is built into the front of the torso 104, corresponding to the chest.

[0038] Figure 4 is a block diagram showing the avatar robot controller 122, which is built into the avatar robot 100 and controls the actions (such as speech and gestures) of the avatar robot 100 based on commands from the operator terminal 14 (described later) shown in Figure 5.

[0039] Referring to Figure 4, the avatar robot controller 122 includes a main control unit 124, which includes a CPU 126, and a memory 130 is connected to the CPU 126 via an internal bus 128. The memory 130 includes ROM and RAM and stores control data and script data (audio files) sent from the operator terminal 14.

[0040] Furthermore, programs necessary for controlling the avatar robot 100 (such as the OS and information acquisition (sensor signal acquisition) programs) are stored in memory 130. The avatar robot controller 122 controls the operation of the avatar robot 100 according to the programs stored in memory 130, in accordance with commands from the local terminal.

[0041] The CPU 126 is further connected to an input interface 132, an output interface 134, a communication unit 136, and an actuator control board 138 via a bus 128.

[0042] The input interface 132 inputs the video signal from the camera 116 and the audio signal from the microphone 120 to the CPU 126 (memory 130).

[0043] Furthermore, although not shown in the figures, the avatar robot 100 is equipped with pressure sensors in required locations, which can detect human contact (weak or strong contact) to those locations. Such pressure sensors 140 can be, for example, a piezoelectric pressure sensor (a sensor with a piezoelectric sensor sheet between two silicon layers) or a cloth-type sensor such as the one disclosed in Japanese Patent Application Publication No. 2021-135169. Either type of pressure sensor outputs a pressure signal (voltage signal) of a magnitude corresponding to the contact pressure. The pressure signal from the pressure sensor 140 is input to the CPU 126 (memory 130) via the input interface 132.

[0044] A speaker 118 is connected to the output interface 134, and the voice of operator OP or synthesized voice from the avatar robot 100 is output from the speaker 118.

[0045] The communication unit 136 includes, for example, a network interface controller (NIC), and the CPU 126 can communicate with the operator terminal 14, an external computer, or other avatar robots 100 (neither of which are shown) via this communication unit 136, and exchange data between them.

[0046] Furthermore, the actuator control board 138 is composed of, for example, a DSP, and controls the operation of the actuators provided in each of the above-mentioned parts of the avatar robot 100, as described below.

[0047] The movement of the two degrees of freedom of the fuselage 104 is controlled by the CPU 126 controlling the fuselage actuator 142 via the actuator control board 138.

[0048] The two degrees of freedom of the right arm 106R and the left arm 106L are controlled by the CPU 126 controlling the arm actuators 144 via the actuator control board 138.

[0049] The three degrees of freedom of the neck, or head 108, are controlled by the CPU 126 via the actuator control board 138 and the head actuator 146.

[0050] The opening and closing movements of the right eye 110R and the left eye 110L, i.e., the eyelids, are controlled by the CPU 126 controlling the eyelid actuators 148 via the actuator control board 138.

[0051] The three degrees of freedom of the eyeballs 112R and 112L are controlled by the CPU 126 controlling the eyeball actuators 150 via the actuator control board 138.

[0052] The opening and closing operation of the mouth 114 is controlled by the CPU 126 controlling the mouth actuator 152 via the actuator control board 138.

[0053] Referring to Figure 5, the operator terminal 14 includes a main control unit 22, which is substantially a computer and includes a CPU 24 and a memory 28 connected to the CPU 24 via a bus 26. The memory 28 includes RAM, ROM, HDD (hard disk drive), etc., although details are not shown. The CPU 24 controls the avatar robot 100 by executing a computer program described later. The above program is pre-stored in ROM or HDD and is loaded into RAM and executed as needed.

[0054] An input interface 30 and an output interface 32 are further connected to bus 26.

[0055] An environmental camera 18, as shown in Figure 2, is connected to the input interface 30, and the video signal from the environmental camera 18 is input from the input interface 30 to the CPU 24 (memory 28).

[0056] The input interface 30 also receives an on / off signal from the emergency switch 20, which is then sent to the CPU 24 (memory 28).

[0057] An input device 38, such as a keyboard or touch panel, is also connected to the input interface 30, and operation signals are input from the input device 38 to the CPU 24 (memory 28).

[0058] The operator terminal 14 is equipped with a microphone 40, and the voice signal from the operator OP via this microphone 40 is input to the CPU 24 (memory 28) through the input interface 30.

[0059] The output interface 32 outputs video signals from, for example, an environmental camera 18 to the display 42 to display the video necessary for the operator, and also outputs audio signals from the main control unit 22 to the speaker 44 to output speech for dialogue with the user USR and speech for deterring infringing acts.

[0060] The communication unit 34 is further connected to the bus 26, and this communication unit 32 communicates with the avatar robot 100, etc. via the network 12. For example, it communicates with data transmitted from the avatar robot 100 (camera signal from camera 116, audio signal from microphone 120). It is used to receive pressure signal data from the pressure sensor 140 and to send setting data for infringement deterrence operations to the avatar robot 100.

[0061] The avatar robot interface 36 transmits commands containing this configuration data to the avatar robot controller 122 (Figure 3), causing the avatar robot controller 122 to execute verbal and non-verbal actions to deter infringing behavior from each avatar robot 100.

[0062] However, if the avatar robot controller 122 of the avatar robot 100 has a communication unit 136 as shown in Figure 4, the communication between the avatar robot 100 and the operator terminal 14 can also be performed by this communication unit 136. In this case, commands may be sent directly from the main control unit 22 to the avatar robot controller 122 without going through the avatar robot interface 36.

[0063] Figure 6 is an illustrative diagram showing an example of the memory map of the RAM of memory 28 in the embodiment shown in Figure 5. As shown in Figure 6, the RAM includes a program storage area 46 and a data storage area 48. The control program for the operator terminal 14 is stored in the program storage area 46.

[0064] The control program for the operator terminal 14 includes an operation detection program 50a, a display control program 50b, a data acquisition program 50c, an infringement determination program 50d, an infringement deterrence program 50e, and the like.

[0065] The operation detection program 50a is a program that detects operations performed on the input device 38 of the operator terminal 14.

[0066] The display control program 50b is a program that displays images on the display 42 (Figure 5).

[0067] The data acquisition program 50c is a program that acquires data such as the video signal from the camera 116, the audio signal from the microphone 120, the pressure signal from the pressure sensor 140, the video signal from the environmental camera 18, and the on / off signal from the emergency switch 20, which are sent from the avatar robot 100 via the communication unit 34, and stores them in the acquired data storage area 52a, which will be described later. However, all of these signals are digitized and stored as digital data. The infringement determination program 50d is a program that determines whether an infringing act has been committed against the avatar robot 100, based on the information or data acquired by the acquisition program 50c.

[0068] For example, an actor is asked to present an infringing act against the avatar robot 100 in advance. At that time, multimodal sensor data (information) is acquired, such as a sequence of camera images (RGBD) from an environmental camera 18 that includes the area around the avatar robot 100 in its field of view, a sequence of audio recorded by a microphone 120 that can record sounds around the avatar robot 100, and a sequence of contact information detected by a pressure sensor 140 installed on the body of the avatar robot 100. Then, the presented infringing act and the information on its severity are linked (paired), and supervised or semi-supervised learning is performed using a deep neural network or a classification tree such as XGBoost (product name) to model the relationship between the paired information and store it as reference data in the judgment data storage area 52b described later.

[0069] Then, when the aforementioned multimodal sensor data (information) acquired by the data acquisition program 50c is input, the infringement determination program 50d can determine how closely the event resembles a pre-defined infringing act, and estimate its severity.

[0070] However, the input to machine learning models such as deep neural networks and XGBoost used to formulate judgment data does not need to be raw data from multimodal sensors. It may be image recognition results from image input, speech recognition results from voice input, or information linguized using a large-scale language model that takes speech recognition results or other multimodal data as input.

[0071] For example, if the avatar robot 100 is in the form of a woman, an infringing act such as attempting to touch the avatar robot 100's breasts can be detected by the aforementioned machine learning model when the camera captures a change in the perpetrator's body area as it approaches the avatar robot 100's body area, or when contact information is captured by a touch sensor installed on the chest area.

[0072] Furthermore, in cases of abusive behavior directed at Avatar Robot 100, similar determinations can be made by capturing abnormal values ​​in audio features such as high sound pressure and pitch as audio data, or by capturing them as content information in the string of text resulting from speech recognition.

[0073] However, the infringing acts presented in this learning process do not have to be limited to those actually acquired by sensors placed near the avatar robot 100 equipped with this function. They may also be video or audio data of scenes depicting infringing acts against robots or humans in movies or news footage, or text data in novels that describe infringing acts.

[0074] In this embodiment, the infringement detection program 50d can classify infringing acts against the avatar robot 100 into categories such as verbal abuse and physical assault and physical contact (including sexual contact), and within each category, the severity is classified into three levels.

[0075] Furthermore, this infringement detection program 50d can also determine that an infringing act has occurred if an ON signal is received from the emergency switch 20 shown in Figure 2.

[0076] The infringement deterrence program 50e is a program that causes the avatar robot 100 to take actions to deter infringement, depending on the category of infringement and its severity determined by the determination program 50d.

[0077] For example, if the Avatar Robot 100 were to attempt to deter the offender through vocalizations, possible responses could include: restraints such as "Please stop," requests for help such as "Someone help me," warnings such as "This is being recorded," warnings such as "I'll report this to the police," advice such as "People are watching," direct references to the determination of an infringing act such as "You touched my chest" or "Your statement that is an infringing act," and indirect references such as "It is not good to touch a woman's body" or "Even robots get hurt when they are verbally abused."

[0078] For example, depending on the severity of the situation, the avatar robot 100 can be made to utter one or more of these phrases.

[0079] If personal information has been obtained, it is thought that stating that the individual can be identified, such as "Mr. / Ms. [Name], please stop," can serve as an additional deterrent.

[0080] If the goal is to deter the avatar robot 100 using its facial expressions, then the avatar robot 100 could display expressions such as anger, confusion, or sadness. For example, depending on the severity of the situation, the avatar robot 100 could display one of these expressions.

[0081] To deter the Avatar Robot 100, possible actions include covering the face with both hands and turning away, crossing arms to convey anger, crossing arms or lowering the head to appear defensive, thrusting both hands towards the opponent to stop them, making an "X" with both hands or fingers, and closing the eyes or stopping to indicate that the opponent is not interested.

[0082] For example, depending on the severity of the situation, the avatar robot 100 can be made to perform one or more of these actions. Although not shown in the diagram, the program storage area 46 also stores other programs necessary for controlling the operator terminal 14.

[0083] The data storage area 48 is configured with an acquired data storage area 52a, a determination data storage area 52b, a suppression setting data storage area 52c, and the like.

[0084] The acquired data storage area 52a is an area for storing the data acquired by the data acquisition program 50c described above.

[0085] The determination data storage area 52b is an area for storing the above-mentioned reference data, which the infringement determination program 50d uses to determine whether or not an infringing act has occurred based on the acquired data.

[0086] The deterrent setting data storage area 52d is a storage area for storing deterrent setting data that the infringement deterrent program 50e transmits to the avatar robot 100 in order to deter infringement by the avatar robot 100.

[0087] Furthermore, the data storage area 48 stores other data necessary for executing the control program of the operator terminal 14, as well as flags and other counters (timers) necessary for executing the control program.

[0088] Figure 7 is a memory map of the memory (RAM) 130 of the avatar robot controller 122 shown in Figure 4. The memory 130 includes a program storage area 154 and a data storage area 156.

[0089] The program storage area 154 stores programs for controlling the operation of the avatar robot 100. In this embodiment, these control programs include, in particular, a data acquisition program 158a and a communication program 158b. The data acquisition program 158a acquires the video signal from the camera 116, the audio signal from the microphone 120, and the pressure signal from the pressure sensor 140, and stores them as digital data in the acquired data storage area 160a of the data storage area 156.

[0090] The communication program 158b exchanges data and signals with the operator terminal 14 via the network 12 (Figure 1). For example, it transmits data acquired by the data acquisition program 158a to the operator terminal 14, or receives configuration data for deterring infringing activities from the operator terminal 14.

[0091] The data storage area 156 includes an acquired data storage area 160a for storing the acquired data described above, as well as a suppression setting data storage area 160b for storing the suppression setting data received from the operator terminal 14, and the like.

[0092] In addition, the data storage area 156 stores other data necessary for executing the control program of the avatar robot 100, as well as flags and other counters (timers) necessary for executing the control program.

[0093] Figure 8 is a flowchart showing an example of the operation of the operator terminal 14 in the embodiment. It should be noted in advance that the operations according to the flowchart in Figure 8 are executed repeatedly at a predetermined period (for example, a frame period), similar to the operations according to the flowchart in Figure 9.

[0094] In the first step S1 of Figure 8, the main control unit 22, or CPU 24, receives and acquires video signal data from camera 116, audio signal data from microphone 120, and pressure signal data from pressure sensor 140 sent from the avatar robot 100, in accordance with the data acquisition program 50c, and also acquires video signal data from environmental camera 18 and on / off signal data from emergency button 20.

[0095] These acquired data are stored in the acquired data storage area 52a for use in the determination of the infringement determination program 50d.

[0096] For the sake of explanation, Figure 8 shows that data is acquired in step S1, but please note that data acquisition by the operator terminal 14 is performed continuously.

[0097] Then, in the next step S3, the CPU 24 determines whether there has been an infringement against the avatar robot 100 based on the various data stored in the acquired data storage area 52a, in accordance with the infringement determination program 50d described above.

[0098] The criteria for making such a determination have already been explained in detail, so I will omit repeating them here. However, if it is determined that an infringing act has occurred, it is necessary to determine whether it falls under the verbal category (such as verbal abuse) or the physical category (such as striking or touching, including sexual harassment), and to what level of severity it is.

[0099] In step S5, the CPU 24 determines, based on the result of that determination, whether there has been an infringing act against the avatar robot 100. It also determines that there has been an infringing act if there is data of an ON signal from the emergency button 20.

[0100] If "NO" is determined in step S5, the process shown in Figure 8 ends there.

[0101] If “YES” is determined in step S5, the CPU 24 sends deterrent setting data to the avatar robot 100 in the next step S7, which causes the avatar robot 100 to perform the deterrent actions described in detail above, in accordance with the deterrent program 50e.

[0102] For example, if you want to deter abusive behavior by having the avatar robot 100 make a sound (language category), you can send sound data such as "Stop it" or "Someone help me" to the avatar robot 100.

[0103] In this case, depending on the severity of the infringing act, it is possible to select voice data with a strength that will be conveyed to the person committing the infringing act, or it is also possible to send only one type of voice data.

[0104] Furthermore, depending on the severity, speech data may be set to cause the avatar robot 100 to produce one or more of these sounds.

[0105] For example, if the goal is to deter abusive behavior through the actions (body category) of the avatar robot 100, setting data is sent to the avatar robot 100 that allows it to perform actions such as "covering its face with both hands and turning its face away" or "crossing its arms and showing anger." The avatar robot 100 then sets the received deterrent setting data on the actuator control board 138 (Figure 4). Accordingly, the corresponding actuators 142-152 are controlled, and the avatar robot 100 can execute deterrent actions according to that setting data.

[0106] In this case as well, you can select the strength of the action that is communicated to the infringer according to the severity of the infringement, or you can simply send only one type of setting data.

[0107] Furthermore, depending on the severity, actuator control data can be set to cause the avatar robot 100 to perform one or more of these body movements.

[0108] As shown in Figure 9, in the avatar robot 100, when it is determined in step S21 that there is a data request from the operator terminal 14, in step S23 the communication unit 136 transmits the video signal data from the camera 116, the audio signal data from the microphone 120, and the pressure signal data from the pressure sensor 140 to the operator terminal 14.

[0109] Then, in step S25, it is determined whether deterrence setting data has been received from the operator terminal 14. If the answer is "YES", in step S27, the CPU 126 sets the received deterrence setting data on the actuator control board 138 (Figure 4), and in step S29, it performs a deterrent action by vocalization and / or physical movement.

[0110] It should be noted that the infringing acts and deterrent actions (voice, body movements) described above are merely examples, and the embodiments described herein may also be applicable to other infringing acts and deterrent actions (voice, body movements).

[0111] Mild deterrent actions include emitting sounds resembling tongue clicks or sighs, changing the Avatar Robot 100's facial expression to convey anger or disgust, and even having the Avatar Robot 100 express refusal or intimidation with an intimidating voice (for example, a deep voice) different from the voice it normally uses during conversations, or making the Avatar Robot 100 perform gestures (body movements) that indicate refusal or intimidation.

[0112] Furthermore, if the infringing acts against the avatar robot 100 do not cease even after the avatar robot 100 has taken these deterrent actions, it may be possible to implement stronger deterrent actions, such as lighting up the area around the avatar robot 100, generating a buzzer sound, and in some cases notifying a third party with deterrent power such as a security company, or notifying users connected to the avatar robot on social media via the network.

[0113] In the above-described embodiment, a three-dimensional robot was used as the avatar robot 100, but the embodiments of this disclosure are also applicable to two-dimensional avatar robots. For example, if we consider an avatar robot dialogue system that displays a two-dimensional avatar robot on a touch display, any unnecessary or disruptive touch operation on that touch display can be treated as an "infringing act" as described above.

[0114] Then, as described above, deterrent actions to deter infringement through voice and / or physical movements can be performed by such a two-dimensional avatar robot.

[0115] Furthermore, in the above-described embodiment, the operator terminal 14 collects various data to determine whether an infringing act has been committed against the avatar robot 100, and based on that, the operator terminal 14 controls the avatar robot 100 to perform infringement deterrent actions.

[0116] However, if, for example, a program performing the same functions as the data acquisition program 50c, infringement detection program 50d, and infringement prevention program 50e shown in Figure 6 is set in the memory 130 of the main control unit 124 of the avatar robot 100, the avatar robot 100 can autonomously perform actions such as determining whether an infringement has occurred and preventing infringement.

[0117] In the above embodiment, the determination means for determining whether an infringing act has occurred uses video signals from an environmental camera 18 that includes the area around the avatar robot 100 in its field of view, video signals from the camera 116 of the avatar robot 100, audio signals recorded by a microphone 120 that can record sounds around the avatar robot 100, and pressure signals of contact information detected by a pressure sensor 140 installed on the body of the avatar robot 100. However, the determination means may determine whether an infringing act has occurred based on at least one or two of these video signals, audio signals, and pressure signals. [Explanation of symbols]

[0118] 10. Avatar robot dialogue system 14... Operator terminal 18…Environmental camera 20…Emergency button 24, 126 ...CPU 100... Avatar Robot 122... Avatar Robot Controller 116 ... Camera 120... Mike 140 ... pressure sensor 138 ... Actuator control board

Claims

1. An avatar robot dialogue system in which a user interacts with an avatar robot controlled by an operator terminal operated by an operator, A determination means for determining whether there is an infringing act against the avatar robot, and An avatar robot dialogue system comprising a means for executing deterrent actions to prevent the aforementioned infringing act when it is determined that such an infringing act has occurred.

2. The system comprises at least one of the following: a camera for photographing the area around the avatar robot, a microphone for acquiring sound from the area around the avatar robot, and a pressure sensor for detecting contact with the avatar robot. The avatar robot dialogue system according to claim 1, wherein the determination means determines whether or not an infringing act has occurred based on at least one of the video signal from the camera, the audio signal from the microphone, and the pressure signal from the pressure sensor.

3. The avatar robot dialogue system according to claim 1, wherein the deterrent action execution means includes at least one of a vocalization means for causing the robot to make a sound in order to deter the infringing act and a bodily movement means for causing the robot to perform a bodily movement in order to deter the infringing act.

4. The avatar robot dialogue system according to claim 1, wherein the determination means includes a severity determination means for determining the severity of an infringing act, and the deterrent action execution means causes the deterrent action to be carried out according to the severity.

Images