Behavioral pattern estimation model and robot equipped with the model

Abstract

This involves using sensors and devices other than contact sensors to estimate human physical actions towards robots. [Solution] In a trained model 100 for estimating patterns of human physical actions toward robot 10, the parameters of the trained model are learned using training data that associates acoustic features generated by human physical actions toward robot 10, movement features of robot 10 caused by human physical actions toward robot 10, and label information of patterns of human physical actions toward robot 10. The computer is configured to accept acoustic features generated by human physical actions toward robot 10 and movement features of robot 10 caused by human physical actions toward robot 10 as input, perform parameter-based calculations on the input acoustic features and movement features, and output patterns of human physical actions toward robot 10.

Description

Technical Field

[0001] The present invention relates to robots designed on the premise of human interaction, such as pet robots and therapy robots, and particularly relates to a technique for estimating patterns of human physical actions performed on such robots.

Background Art

[0002] In recent years, research and development of pet robots and therapy robots for healthcare and communication purposes have been actively carried out. These pet robots perform communication by detecting human contacts such as conversation and physical contact using a large number of sensors including microphones, cameras, tactile sensors, and acceleration sensors (for example, refer to Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0004] To understand diverse human contacts, it is necessary to install a large number of various sensors, but some high-performance sensors cost tens of thousands of yen each, which causes an increase in the cost of the robot. In particular, contact sensors for detecting that a human has touched the robot need to be arranged at a plurality of positions on the robot, which has a significant impact on cost.

[0005] Accordingly, an object of the present invention is to provide a mechanism capable of detecting human physical actions on a robot using sensors and devices originally intended for other purposes than contact sensors, and a robot equipped with such a mechanism.

Means for Solving the Problems

[0006] According to one aspect of the present invention, a trained model is provided for estimating patterns of physical human actions toward a robot, wherein the parameters of the trained model are trained using training data that associates acoustic features generated by the physical human actions toward the robot, features of the robot's movement caused by the physical human actions toward the robot, and label information of patterns of physical human actions toward the robot. The trained model is configured to cause a computer to accept acoustic features generated by the physical human actions toward the robot and features of the robot's movement caused by the physical human actions toward the robot as input, perform calculations based on the parameters on the input acoustic features and features of the movement, and output patterns of physical human actions toward the robot.

[0007] According to another aspect of the present invention, a robot is provided which has a microphone and a camera built into it, and which has a controller that performs the above-described behavior pattern estimation model. The controller performs a first step of processing acoustic data picked up by the microphone to extract the acoustic features, and a second step of processing video data captured by the camera to extract the motion features.

[0008] According to another aspect of the present invention, a robot is provided which incorporates a microphone and an acceleration sensor and includes a controller that performs the above-described behavior pattern estimation model. The controller performs a first step of processing acoustic data picked up by the microphone to extract acoustic features, and a second step of processing measurement data from the acceleration sensor to extract motion features. [Effects of the Invention]

[0009] According to the present invention, physical actions by a person towards a robot can be detected using sensors or devices intended for purposes other than contact sensors. This makes it possible to detect and identify physical actions by a person towards a robot without equipping the robot with contact sensors, thereby reducing the cost of the robot. [Brief explanation of the drawing]

[0010] [Figure 1] This is a system configuration diagram of a robot according to one embodiment of the present invention. [Figure 2] This is a schematic diagram of a trained model that estimates patterns of human physical actions towards robots. [Figure 3] This figure shows examples of acoustic spectrograms observed in various behavioral patterns. [Figure 4] This table shows examples of acoustic features used for estimating behavioral patterns. [Figure 5] This figure shows examples of the 3-axis acceleration of a robot observed in various action patterns. [Figure 6] This is a table of F1 scores for behavioral pattern recognition by feature. [Figure 7] This figure shows the confusion matrix for behavioral pattern recognition using video features. [Figure 8] This figure shows the confusion matrix for behavior pattern identification using acceleration features. [Figure 9] This figure shows the confusion matrix for behavior pattern recognition using acoustic features. [Figure 10] This figure shows the confusion matrix for behavior pattern recognition using video and audio features. [Figure 11] This figure shows the confusion matrix for behavior pattern recognition using video features, acoustic features, and acceleration features. [Modes for carrying out the invention]

[0011] The embodiments of the present invention will be described in detail below, with reference to the drawings as appropriate. However, unnecessary details may be omitted. For example, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily verbose and to facilitate understanding by those skilled in the art. The inventors provide the accompanying drawings and the following explanation so that those skilled in the art can fully understand the present invention, and do not intend to limit the subject matter described in the claims by means of these.

[0012] [Robot system configuration] The robot according to the present invention is a robot that users touch and use, such as a pet robot or a therapy robot. The size and weight of the robot are such that a user can place it on their lap or pick it up. Since it is intended to be touched by a user, the surface of the robot is assumed to be made of a soft, cushioning material such as cloth or yarn, but this is not the only requirement.

[0013] Figure 1 is a system configuration diagram of a robot according to one embodiment of the present invention. The robot 10 according to this embodiment includes, as its main components, a microphone 11, a camera 12, an acceleration sensor 13, and a microcontroller 14.

[0014] Microphone 11 is for capturing external sounds from the robot 10, particularly the user's voice. Microphone 11 is mounted in a location that can easily pick up the user's voice, such as the front of the robot 10. Camera 12 is for capturing images of the area around the robot 10, particularly the user's facial expressions, and generates video data at, for example, 30 FPS. Camera 12 is mounted in a location that can capture the user's face, such as the front of the robot 10. Microphone 11 and camera 12 may be integrated into a single unit.

[0015] The acceleration sensor 13 is embedded inside the robot 10. As the acceleration sensor 13, for example, a 6-axis attitude sensor equipped with a 3-axis gravitational accelerometer and a 3-axis gyroscope can be used. The acceleration sensor 13 measures the acceleration of each axis at a predetermined sampling cycle (for example, 0.1 seconds) when the robot 10 receives an external force such as from a user touching the robot 10 and its attitude changes.

[0016] The microcomputer 14 is a compact computer system integrating a processor, a work memory (volatile memory), a non-volatile memory, a timer, an A / D converter, an I / O interface, and the like. Signals from the microphone 11, the camera 12, and the acceleration sensor 13 are collected and processed by the microcomputer 14. Specifically, the microcomputer 14 extracts various feature quantities from the respective signals of the microphone 11, the camera 12, and the acceleration sensor 13 and inputs them to a trained model, causing the trained model to estimate patterns of various physical actions of a person on the robot 10. The action patterns to be estimated are actions in a broad sense that include not only contact actions such as "stroking" and "hitting", but also the motion / action of "picking up" and the "stationary" state where no action is performed.

[0017] [Action Pattern Estimation Model] Figure 2 is a schematic diagram of a trained model that estimates patterns of a person's physical actions on a robot. The trained model 100 is stored as parameters in the non-volatile memory of the microcomputer 14. The processor of the microcomputer 14 reads the parameters of the trained model 100 from the non-volatile memory and loads them into the work memory, so that the microcomputer 14 functions as the trained model 100.

[0018] The trained model 100 receives acoustic features extracted from the sounds generated by a person's physical actions towards the robot 10 and motion features extracted from the robot 10's movements due to the person's physical actions towards the robot 10. It then performs calculations on these multimodal features based on trained parameters to estimate and output patterns of physical actions by a person towards the robot 10. While various action patterns are possible to output, for convenience, we will estimate four patterns here: "still," "stroking," "tapping," and "picking up."

[0019] Acoustic features are extracted by processing the acoustic data captured by the microphone 11. Motion features encompass both video features extracted by processing video data captured by the camera 12 and acceleration features extracted by processing measurement data from the acceleration sensor 13. Details of the features will be described later.

[0020] Action patterns can be estimated from the features of video, acceleration, and sound. Since video and acceleration are time-series data, a neural network suitable for learning time-series data, such as LSTM (Long Short-Term Memory), is used to handle these features. On the other hand, as will be discussed later, sound features are static data, so a neural network suitable for learning static data, such as LightGBM (Light Gradient Boosting Machine), is used to handle sound features.

[0021] Alternatively, behavioral patterns can be estimated from multimodal features, such as video + audio features, or video + acceleration + audio features. In this case, the dynamic video and acceleration features can be converted into static features, matched with the sampling period of the audio features, and input into LightGBM as static data along with the audio features.

[0022] [Feature extraction] The microcontroller 14 extracts various features to be input to the trained model 100 using, for example, the following method.

[0023] When a user touches the robot 10, vibrations caused by the physical contact between the user's body and the surface of the robot 10 are transmitted to the surrounding air, generating sound. The microcontroller 14 obtains an acoustic spectrogram from the acoustic signal picked up by the microphone 11 and extracts acoustic features at a predetermined sampling period (e.g., 1 second).

[0024] Figure 3 shows examples of acoustic spectrograms observed in various action patterns. (a) is an example of an acoustic spectrogram generated when a person "strokes" the robot 10, (b) is an example of an acoustic spectrogram generated when a person "hits" the robot 10, and (c) is an example of an acoustic spectrogram generated when a person "picks up" the robot 10. From these graphs, it can be seen that the characteristics of the acoustic spectrograms generated from the robot 10 differ depending on the action pattern.

[0025] As a tool for extracting acoustic features, one can use, for example, openSMILE, open-source software developed by the Technical University of Munich and audEERING. OpenSMILE is a tool for analyzing, processing, and classifying speech, i.e., human voices, but it can also extract acoustic features themselves, such as fundamental frequency and loudness.

[0026] In estimating patterns of physical human actions towards robot 10, the acoustic features generated when a person takes an action towards robot 10 are considered more important than the features of the user's voice. Therefore, the acoustic features related to human voice obtained using openSMILE do not necessarily need to be used, and in this embodiment, the features related to human voice are excluded. As a result, the number of acoustic features is reduced, allowing learning to be performed with a smaller amount of training data.

[0027] Figure 4 is a table illustrating acoustic features used for behavior pattern estimation. As an example, as shown in the table in Figure 4, 37 of the 88 special features included in the eGeMAPSv02 feature set, excluding those related to speech, can be used. In the table, amean is the arithmetic mean, stddevNorm is the standard deviation divided by the arithmetic mean, percentile20.0(50.0, 80.0) is the 20% (50%, 80%) value from the top, pctlrange0-2 is the range between percentile20.0 and 80.0, meanRisingSlope is the mean slope in the part where the signal is increasing, stddevRisingSlope is the standard deviation of the slope in the part where the signal is increasing, meanFallingSlope is the mean slope in the part where the signal is decreasing, and stddevFallingSlope is the standard deviation of the slope in the part where the signal is decreasing.

[0028] When a person performs an action on robot 10, an external force is applied to robot 10, causing its position and orientation to change. The microcontroller 14 extracts movement characteristics of robot 10 caused by the person's physical action from the video data captured by camera 12 and / or the measurement data from acceleration sensor 13.

[0029] Figure 5 shows examples of the three-axis acceleration of a robot observed in various action patterns. (a) is a graph of the acceleration of robot 10 when a person "strokes" robot 10, (b) is a graph of the acceleration of robot 10 when a person "hits" robot 10, and (c) is a graph of the acceleration of robot 10 when a person "picks up" robot 10. The parts where the waveform moves significantly represent the fluctuations in acceleration when various actions occur. Note that, due to the specifications of the acceleration sensor 13 used, the reference value for the stationary state of the Y axis is different from the reference values ​​for the other two axes, so the Y-axis acceleration is measured with a deviation from the acceleration of the other two axes. From these graphs, it can be seen that the characteristics of the three-axis acceleration of robot 10 also differ depending on the action pattern.

[0030] The acceleration feature includes the acceleration of each x, y, and z axis at a predetermined sampling period obtained from the measurement data of the acceleration sensor 13, and may also include the difference between the x, y, and z axis accelerations and the three-axis composite acceleration.

[0031] On the other hand, video features are extracted from video data captured by camera 12. As a tool for extracting video features, for example, the open-source MediaPipe developed by Google can be used. The microcontroller 14 can use MediaPipe to, for example, track the position (position on the x and y axes) of the user's nose landmark in the video during the action and extract this as a video feature. In addition to the position of the landmark on each x and y axis obtained from the video data of camera 13, the video features may also include the coordinate difference and the coordinate average over a predetermined number of frames (e.g., 3 frames).

[0032] [Evaluation experiment] An experiment was conducted using a stuffed animal incorporating a microphone, camera, and accelerometer, which was treated as robot 10. This experiment focused on four behavioral patterns. Participants were instructed to perform each of the following actions—stroking, tapping, and picking up—for at least 3 seconds each, in any order, with at least 3 seconds of stillness in between. For "picking up," participants were instructed to lift and lower the stuffed animal until their face was out of the camera's field of view. The stuffed animal was positioned in front of the participant so that their entire face was visible in the frame. Participants touched the stuffed animal in a one-on-one face-to-face manner, ensuring they did not obstruct the camera with their hands or arms. They were also instructed not to make any noise, and data was collected in a quiet environment.

[0033] The experiment involved four male participants aged 22 and 24. Three of them were instructed to touch a stuffed animal with two different levels of pressure: a natural touch and a firm touch, and data was collected five times for each. The remaining participant was instructed to touch the stuffed animal with a firm touch three times. Of the 33 data points collected from these experiments, six were unusable due to excessive screen shake and four due to high microphone noise, so the experiment was conducted using the remaining 23 data points.

[0034] Furthermore, while ambient sounds such as group conversations and everyday noises were played through a speaker, one participant was instructed to touch a stuffed animal in two different intensity patterns, and the sounds of each action were collected five times.

[0035] The following table shows the number of feature samples for each behavior pattern. Video features consist of the six types mentioned above, acoustic features of the 37 types mentioned above, and acceleration features of the 7 types mentioned above. Features were calculated in units of 1 / 30th of a second, 1 second, and 0.1 seconds, respectively. Each measurement data is approximately 37 seconds long, and since 23 measurement data are used, there are several thousand to tens of thousands of feature samples. Table: Number of feature samples for each behavior pattern TIFF2026168787000002.tif25151

[0036] Furthermore, since the trends in behavioral data are thought to differ depending on the participant and the strength of touch, in the machine learning experiment, measurement data of different contact strengths from the same participant were treated as separate data, and the experiment was conducted using leave-one-out cross-validation.

[0037] [Experimental conditions] The LSTM parameters were set as follows. ■ Dimensionality of the LSTM layer output: 100 ■Activation function: ReLU ■Dropout:0.1 ■Fully connected layer: 4 ■Activation function: softmax ■Time step: 30 (video features), 5 (acceleration features) ■Batch size: 6 ■ Number of epochs: 300 ■Loss function: Cross-entropy error ■Optimization function: Adaptive moment estimation (ADAM)

[0038] The LightGBM parameters were set as follows. Optuna, developed by Preferred Networks, was used for hyperparameter tuning. ■Optuna trial count: 50 ■num_locals:3~63 ■max_depth:2~13 ■min_data_in_leaf:1~30 ■learning_rate:0.01~0.1

[0039] In this experiment, we conducted comparative experiments using a total of five feature sets: a subset of standalone video features, a subset of standalone acoustic features, a subset of standalone acceleration features, a feature subset combining video and acoustic features, and a feature subset combining video, acoustic, and acceleration features. The standalone features are as described above. Video and acceleration features were trained as dynamic features, while acoustic features were trained as static features.

[0040] Regarding multimodal features, the video, acoustic, and acceleration features have different feature calculation times (1 / 30 second (1 frame), 0.1 seconds, and 1 second), respectively. Therefore, time synchronization is required, and the acoustic features, which have the longest time, are used in synchronization. The video features are synchronized by using the average and maximum coordinates of the nose landmark over 1 second, and the acceleration features are synchronized by using the average, standard deviation, maximum, and minimum acceleration values ​​in the x, y, and z axes over 1 second. These are then converted into static features for training.

[0041] Furthermore, by calculating the aforementioned acoustic features from acoustic data collected in the presence of ambient noise and training them together with a subset of single acoustic features, we compare the recognition accuracy in the presence of ambient noise.

[0042] [Experimental Results] Figure 6 is a table of F1 scores for behavior pattern recognition by feature. Figures 7 through 11 show the confusion matrices for behavior pattern recognition based on video features, acceleration features, acoustic features, video features and acoustic features, and video features, acoustic features, and acceleration features, respectively.

[0043] When trained individually, the acoustic feature subset trained with LightGBM had the highest overall F1 score at 84%, followed by the video feature subset trained with LSTM at 79%, and the acceleration feature subset at 71%. Regarding the acoustic feature subset, "tapping" had the highest F1 score, while "stroking" and "lifting" had similar F1 scores. "Stroke" and "tapping" were misidentified to a similar degree. Regarding the video feature subset, "lifting" had the highest F1 score, excluding "still." "Tapping" was misidentified to a similar degree as "still" and "stroking." Regarding the acceleration feature subset, the overall F1 score was low at 71%, with "tapping" being particularly low at 54%. Both the video feature subset and the acceleration feature subset strongly misidentified "tapping" as "still." The results showed that "stationary" consistently yielded the highest F1 score across all three feature subsets, indicating that the three behavior patterns were misrecognized by each other.

[0044] When trained with multimodal feature subsets, the feature subset combining video, audio, and acceleration features showed a higher F1 score than the feature subset combining video and audio features. The feature subset combining video and audio features also showed a high F1 score. In both feature combinations, the three behavior patterns were not misidentified as "stationary," but each was misidentified by the others.

[0045] The following table shows the F1 score and recall of the four behavior patterns when LightGBM was trained using acoustic features of situations where environmental sounds such as group conversations and everyday noises were present during actions involving stuffed animals. Table: F1 score and recall for behavioral pattern identification using acoustic features with and without ambient sound. TIFF2026168787000003.tif32156

[0046] Compared to acoustic features in the absence of ambient noise, the F1 score for acoustic features in the presence of ambient noise was lower, but the overall F1 score was a relatively high 81%. Similar to the acoustic features in the absence of ambient noise, "petting" and "picking up" were misrecognized by each other.

[0047] [effect] As described above, according to this embodiment, by using the microphone 11, camera 12, and acceleration sensor 13 individually or in combination, it is possible to detect a person's physical actions toward the robot 10. This makes it possible to detect and identify a person's physical actions toward the robot 10 without equipping the robot 10 with a contact sensor, thereby reducing the cost of the robot 10.

[0048] [Differentiation] The behavioral patterns estimated by the trained model 100 are not limited to the four: "still," "stroking," "hitting," and "picking up." For example, depending on the training, it can also estimate behavioral patterns such as "shaking," "throwing," "swinging," and "rubbing cheeks."

[0049] Alternatively, motion features of the robot 10 can be extracted from video data that does not show the user's face. For example, using PTAM (Parallel Tracking and Mapping), developed by Oxford University, the movement of camera 12, i.e., the movement of robot 10, can be estimated from video data. In this case, the movement of robot 10 in three-dimensional space can be estimated from the video data and used as video features.

[0050] Since the accuracy of behavior pattern estimation differs depending on the presence or absence of ambient noise, the microcontroller 14 may switch between and use a trained model that estimates behavior patterns from the above-mentioned multimodal features, a trained model that estimates behavior patterns from single acoustic features, and a trained model that estimates behavior patterns from single video features and / or single acceleration features, depending on the ambient noise level. For example, when the ambient noise level is low, the trained model that estimates behavior patterns from single acoustic features may be used to estimate behavior patterns.

[0051] Alternatively, the microcontroller 14 may estimate the behavioral patterns by weighting and adding the outputs of the trained model that estimates behavioral patterns from the multimodal features, the trained model that estimates behavioral patterns from single acoustic features, and the trained model that estimates behavioral patterns from single video features and / or single acceleration features according to the noise level around the robot 10. For example, if the ambient noise level is low, the weight of the output of the trained model that estimates behavioral patterns from single acoustic features may be increased to estimate the behavioral patterns.

[0052] As described above, embodiments have been explained as examples of the technology in the present invention. For this purpose, accompanying drawings and a detailed description have been provided. Therefore, among the components described in the accompanying drawings and detailed description, there may be not only components that are essential for solving the problem, but also components that are not essential for solving the problem, in order to illustrate the above technology. For this reason, the mere fact that these non-essential components are described in the accompanying drawings and detailed description should not be immediately assumed to be essential. Furthermore, since the above embodiments are for the purpose of illustrating the technology in the present invention, various changes, substitutions, additions, omissions, etc., can be made within the scope of the claims or equivalents. [Explanation of symbols]

[0053] 10 Robots 11. Microphone 12 cameras 13. Accelerometer 14 Microcontroller (Controller) 100 pre-trained models

Claims

1. A trained model that estimates patterns of human physical actions towards robots, The parameters of the aforementioned trained model were learned using training data that associates acoustic features generated by a person's physical actions toward the robot, features of the robot's movement caused by a person's physical actions toward the robot, and label information of patterns of the person's physical actions toward the robot. Computers, An action pattern estimation model characterized by receiving acoustic feature quantities generated by a person's physical action toward the robot and motion feature quantities of the robot resulting from a person's physical action toward the robot as input, performing calculations based on the parameters on the input acoustic feature quantities and motion feature quantities, and outputting a pattern of a person's physical action toward the robot.

2. The behavioral pattern estimation model according to claim 1, wherein the features related to human voice are excluded from the aforementioned acoustic features.

3. The behavior pattern estimation model according to claim 1, wherein the motion feature quantities include motion feature quantities in the XYZ directions of the robot.

4. A robot equipped with a microphone and a camera, A controller that performs the behavior pattern estimation model described in any one of claims 1 to 3, The aforementioned controller A first step involves processing the acoustic data picked up by the microphone to extract the acoustic feature quantities, A robot characterized by performing a second step of processing video data captured by the aforementioned camera to extract the characteristic quantities of the motion.

5. A robot equipped with a microphone and an accelerometer, A controller that performs the behavior pattern estimation model described in any one of claims 1 to 3, The aforementioned controller A first step involves processing the acoustic data picked up by the microphone to extract the acoustic feature quantities, A robot characterized by performing a second step of processing the measurement data of the acceleration sensor and extracting the characteristic quantities of the motion.

6. It also incorporates an accelerometer, The aforementioned controller The robot according to claim 4, wherein in the second step, the motion features are extracted by further taking into account the measurement data of the acceleration sensor.

7. A behavior pattern estimation model according to any one of claims 1 to 3 as a first trained model, A second trained model takes acoustic features generated by a person's physical actions toward the robot as input and outputs patterns of the person's physical actions toward the robot, A third trained model takes the characteristic quantities of the robot's movement caused by a person's physical actions toward the robot as input, and outputs the patterns of the person's physical actions toward the robot, It includes a controller that performs the following: The robot is characterized in that the controller switches between the first to third trained models according to the noise level around the robot, or weights and adds the outputs of the first to third trained models according to the noise level around the robot to estimate patterns of physical human actions toward the robot.

Images