Prediction device, subjective impression prediction method, and program

Abstract

To provide a prediction apparatus, a subjective impression prediction method, and a program which predict subjective impression of a dialogue person.SOLUTION: It is known that even in a non-linguistic activity in a dialogue between persons, movement of a head plays various roles. For example, a speaker expresses head movement when he / she emphasizes speech and confirms reaction of a listener, and a listener expresses its head movement as a sign of reply, feedback, and agreement. Thus, the head movement has a plurality of functions, and sometimes a single head movement simultaneously has a plurality of meanings. In a communication system, a prediction apparatus 3 according to the present invention acquires data relating a head movement of a dialogue person from video data obtained by capturing the dialogue person, detects head movement function relative to data relating to the head movement by using a head movement function detection model, calculates a feature amount of the head movement function based on the detection result, and predicts subjective impression of the dialogue person with respect to the feature amount of the head movement function by using a regression model.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a prediction device, a subjective impression prediction method, and a program. [Background technology]

[0002] Head movements are known to play various roles among non-verbal behaviors that occur during human dialogue. For example, speakers use head movements to emphasize their speech or to confirm a response, and listeners use head movements to provide interjections, responses, or signs of agreement. As such, head movements have multiple functions, and one head movement may simultaneously have multiple meanings. Focusing on the multiplicity and ambiguity of the functions of head movements, Non-Patent Document 1 defines functional categories of head movements, and proposes a convolutional neural network (CNN) that uses data that has been non-exclusively annotated to allow overlapping functions and detects the presence or absence of each function from the time series of head posture and the presence or absence of speech. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] K. Otsuka and M. Tsumori, "Analyzing multifunctionality of head movements in face-to-face conversations using deep convolutional neural networks," IEEE Access, vol.8, pp.217169-217195, 2020. Summary of the Invention [Problem to be solved by the invention]

[0004] However, as mentioned above, although head movement function is thought to be deeply related to behavior and emotions during a conversation, there is as yet no established method for predicting the subjective impressions that interlocutors themselves have during a conversation.

[0005] The present invention has been made in view of the above-mentioned points, and has an object to predict the subjective impressions that interlocutors themselves have during a conversation. [Means for solving the problem]

[0006] In order to solve the above problem, the invention according to claim 1 comprises: The interlocutors themselves during the dialogue This prediction device predicts subjective impressions, and is characterized by having: a data acquisition means for acquiring data regarding the head movement of the interlocutor from video data obtained by photographing the interlocutor; a head movement function detection means for detecting head movement function in relation to the data regarding head movement using a head movement function detection model; a feature calculation means for calculating a feature of the head movement function based on the detection result by the head movement function detection means; and a subjective impression prediction means for predicting the interlocutor's subjective impression of the feature of the head movement function using a regression model.

[0007] The invention according to claim 2 is as follows: The interlocutors themselves during the dialogue A prediction device for predicting subjective impressions, comprising: a data acquisition means for acquiring data regarding head movement of the interlocutor from video data obtained by photographing the interlocutor; a head movement detection means for detecting the presence or absence of head movement in relation to the data regarding head movement using a head movement detection model; a head movement function detection means for detecting head movement function in relation to head movement presence or absence information indicating the presence or absence of head movement using a head movement function detection model; a feature calculation means for calculating a feature of the head movement function based on the detection result by the head movement function detection means; and a subjective impression prediction means for predicting the subjective impression of the interlocutor regarding the feature of the head movement function using a regression model.

[0008] The invention according to claim 5 is as follows: The interlocutors themselves during the dialogueA prediction device for predicting a subjective impression, comprising: a data acquisition means for receiving interlocutor data related to an interlocutor and acquiring information on speech sections indicating the interlocutor's speech sections from the interlocutor data; a head movement detection means for detecting the presence or absence of speech in response to the speech section information using a head movement detection model; a head movement function detection means for detecting head movement function in response to speech presence information indicating the presence or absence of speech using a head movement function detection model; a feature calculation means for calculating a feature of the head movement function based on the detection result by the head movement function detection means; and a subjective impression prediction means for predicting the interlocutor's subjective impression of the feature of the head movement function using a regression model. Effect of the Invention

[0009] As described above, the present invention has an effect of making it possible to predict the subjective impressions that interlocutors themselves have during a conversation. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of a communication system. [Diagram 2] FIG. 2 is a diagram illustrating an electrical hardware configuration of the prediction device. [Diagram 3] FIG. 2 is a functional configuration diagram of a prediction device. [Figure 4] FIG. 2 is a diagram showing the relationship between input information (data) and output information (data) in each function. [Diagram 5] 11 is a flowchart showing the behavior of data learning. [Figure 6] 1 is a flowchart illustrating the behavior of data prediction. [Figure 7] FIG. 13 is a diagram showing categories and functional contents of head movement functions. [Figure 8] 1 is a diagram showing categories and functional contents of head movement functions used in this embodiment. FIG. [Figure 9] FIG. 13 is a diagram showing a list of kinematic features. [Figure 10]FIG. 1 illustrates characteristics of head movement function. [Figure 11] FIG. 13 shows predicted values ​​of subjective impressions as introspection report scores for each item. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] [System configuration of the embodiment] First, an outline of the configuration of a communication system according to the present embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram of a communication system according to an embodiment of the present invention.

[0013] 1, a communication system 1 of the present embodiment is constructed by an input device 2, a prediction device 3, and a monitor 4. The input device 2 is a camera 2a, a smartphone 2b, a headphone with a microphone 2c, a microphone 2d, etc. The camera 2a is a luminance camera, a LiDAR (Light Detection and Ranging), an infrared camera, a thermal camera, etc.

[0014] The input device 2 and the prediction device 3 can communicate with each other via a communication network 100 such as the Internet. The communication network 100 may be connected in either a wireless or wired manner.

[0015] The prediction device 3 is configured with one or more computers. When the prediction device 3 is configured with multiple computers, it may be referred to as a "prediction device" or a "prediction system."

[0016] The prediction device 3 predicts the subjective impression (feelings) of the interlocutor based on the interlocutor data on the interlocutor acquired by the input device 2, and outputs result data indicating the prediction result. As an output method, the result data may be transmitted to a monitor 4 as an output device, and a graph or the like related to the result data may be displayed on the monitor 4 side.

[0017] [Hardware configuration] <Hardware configuration of the prediction device> Next, the electrical hardware configuration of the prediction device 3 will be described with reference to Fig. 2. Fig. 2 is a diagram showing the electrical hardware configuration of the prediction device.

[0018] As shown in FIG. 2, the prediction device 3 is a computer and includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, a HD (Hard Disk) 304, a HDD (Hard Disk Drive) controller 305, an external device connection I / F (Interface) 308, a network I / F 309, a bus line 310, and a media I / F 314.

[0019] Of these, the CPU 301 controls the overall operation of the prediction device 3. The ROM 302 stores programs used to drive the CPU 301, such as an IPL (Initial Program Loader). The RAM 303 is used as a work area for the CPU 301.

[0020] The HD 304 stores various data such as programs. The HDD controller 305 controls reading and writing of various data from and to the HD 304 under the control of the CPU 301. Instead of the HD 304 and the HDD controller 305, a solid state drive (SSD) and an SSD controller may be installed.

[0021] The external device connection I / F 308 is an interface for connecting various external devices, such as a display, a speaker, a keyboard, a mouse, a USB (Universal Serial Bus) memory, and a printer.

[0022] The network I / F 309 is an interface for performing data communication via the communication network 100. The bus line 310 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 301 shown in FIG.

[0023] Furthermore, the media I / F 314 controls reading and writing (storing) data from and to a recording medium 313 such as a flash memory. The recording medium 313 includes a DVD (Digital Versatile Disc) and a Blu-ray Disc (registered trademark).

[0024] [Functional configuration of the prediction device] Next, the functional configuration of the prediction device will be described with reference to Fig. 3. Fig. 3 is a diagram showing the functional configuration of the prediction device.

[0025] In Fig. 3, the prediction device 3 has a data acquisition unit 31, a detection unit 32, a feature amount calculation unit 33, a subjective impression prediction unit 34, and a model storage unit 39. The data acquisition unit 31, the detection unit 32, the feature amount calculation unit 33, and the subjective impression prediction unit 34 are functions realized by instructions from the CPU 301 in Fig. 2 based on a program. The data acquisition unit 31 and the detection unit 32 perform data learning and data prediction behavior. The data acquisition unit 31, the detection unit 32, the feature amount calculation unit 33, and the subjective impression prediction unit 34 perform data prediction behavior. In addition, the detection unit 32 includes a head movement detection unit 32a and a head movement function detection unit 32b.

[0026] 2 includes a model storage unit 39. A head movement detection model 40a, a head movement function detection model 40b, and a regression model 50 are stored in the model storage unit 39.

[0027] <Each model> Next, each model will be described.

[0028] (Head movement detection model) Head movement detection model 40a is based on the Late-fusion CNN model (see Non-Patent Document 1) proposed by Otsuka & Tsumori as a CNN (Convolutional Neural Networks) model structure, with a Dropout layer inserted in fc4 of the fully connected layer. Head movement detection model 40a uses the angular velocity of head pose angles (Azimuth, Elevation, Roll) of 32 video frames in video data centered on a target frame as input information (data), and trains CNN to output the presence or absence of head movement for this input information. The number of video frames can be set arbitrarily. Hereinafter, a section of video frames in which head movement is detected continuously is referred to as one "head movement section." Head movement detection model 40a on which machine learning has been performed is stored in model storage unit 39.

[0029] (Head movement function detection model) Head movement function detection model 40b uses a late-fusion CNN model similar to head movement detection model 40a. Here, each video frame in the head movement section detected by head movement detection model 40a is used as the video frame to be predicted, and at least one of the angular velocity of the head posture angle and speech section data for 32 frames centered on each video frame is used as input information. The number of video frames can be set arbitrarily.

[0030] As the speech data, a time series obtained by smoothing a binary time series representing the presence or absence of speech using a moving average filter with a width of 50 video frames before and after is used. The width of this video frame can be set arbitrarily. In the preliminary machine learning, the head movement function detection model 40b is stored in the model storage unit 39.

[0031] FIG. 7 is a diagram showing the categories and function contents of head motor functions. In the pre-learning, a corpus (data in which information on dialogue is written as time-series data) is used to learn a head motor function detection model for one or more of the functions in any category listed in FIG. 7. The corpus includes data in which one or more of the functions in the categories of head motor functions listed in FIG. 7 are non-exclusively labeled. For example, data in which the presence or absence of each of the 32 types of head motor functions listed in FIG. 7 is non-exclusively labeled for each video frame is used for a total of four consensus-building dialogues (total of 27 minutes) between two groups of eight women in their 20s and 30s. However, in this embodiment, an example is described in which machine learning of head motor function detection is performed for each of the 10 types of functions listed in FIG. 8. Note that FIG. 8 is a diagram showing the categories and function contents of head motor functions used in this embodiment. The head motor function detection model 40b on which machine learning has been performed is stored in the model storage unit 39.

[0032] (Regression model) Regression model 50 is a random forest regression model (Reference 1) for predicting introspection report scores as predictors of subjective impressions.

[0033] (Reference 1) L. Breiman, "Random forests," Machine Learning, vol.45, pp.5- 32, 2001. In addition, the regression model 50 is not limited to Random Forest, but may be any model such as Linear Regression or Support Vector Regression.

[0034] A model (MK+F) is constructed by adding at least one of 22 head movement function features to six kinematic features as input information (feature amounts in this case). Then, the regression model 50 performs machine learning of the four introspection report score items, "atmosphere," "fun," "motivation," and "concentration," as output information (correct answer data).

[0035] The number of items in the introspection report score can be set arbitrarily. The introspection report score is data that is inputted and divided at arbitrary time intervals, and the divided intervals are called "slots." The slots are the units of prediction. The length of the intervals may be a uniquely determined value, or the length of the intervals may vary. If the length of the intervals varies, the length of each interval is stored separately in the model storage unit 39.

[0036] <Each function configuration> Next, each functional configuration of the prediction device will be described with reference to FIG.

[0037] The data acquisition unit 31 acquires coordinate information relating to the head movement of the interlocutor from video data obtained by photographing the interlocutor with the camera 2a or the like.

[0038] Furthermore, the data acquisition unit 31 receives interlocutor data on the interlocutor obtained by the camera 2a, etc., and acquires information on speech sections indicating speech sections of the interlocutor from the interlocutor data. When the data acquisition unit 31 acquires information on speech sections, there are the following patterns. (1) The data acquisition unit 31 acquires information about speech sections from audio data included in video data serving as interlocutor data obtained by photographing an interlocutor. (2) The data acquisition unit 31 acquires information about speech sections from voice data serving as interlocutor data obtained by collecting the voices of interlocutors. (3) The data acquisition unit 31 acquires information about a speech section by determining that an utterance has occurred when the motion vector of a feature point existing around the mouth of a conversation partner in the video data is equal to or greater than a predetermined threshold value. (4) The data acquisition unit 31 acquires information about speech sections from a corpus in which speech content is transcribed with time-series information for video data.

[0039] Note that, although the data acquisition unit 31 acquires coordinate information (information on speech intervals) related to head movement in real time after acquiring video data (partner data) from the input device 2, the data acquisition unit 31 may acquire, but not be limited to, the video data (partner data) from the input device 2, temporarily store the data in the RAM 303 or the like, and when the prediction device 3 outputs a predicted value of the subjective impression, the data acquisition unit 31 may read out the video data (partner data) from the RAM 303 or the like.

[0040] Head movement detection section 32a uses the head movement detection model to detect the presence or absence of head movement with respect to the coordinate information related to the head movement. Head movement detection section 32a also uses the head movement detection model to detect the presence or absence of speech with respect to the information on the speech section.

[0041] If a corpus in which dialogue information is transcribed exists, head movement detection unit 32a may output the presence or absence of speech from the corpus. In this case, the process of head movement detection unit 32a is omitted.

[0042] Furthermore, when video data is input as data acquired by data acquisition unit 31, head movement detection unit 32a may extract time-series information on the coordinates located at the head from the video data by using a library such as OpenFace. If the Euclidean distance of movement between certain times in this extracted time-series information exceeds a threshold, it can be determined that head movement has occurred, in which case the processing by head movement detection unit 32a is omitted and data is input directly from data acquisition unit 31 to head movement function detection unit 32b.

[0043] Head movement function detection unit 32b uses the head movement function detection model to detect head movement function in response to head movement presence / absence information indicating the presence or absence of head movement. Head movement function detection unit 32b also uses the head movement function detection model to detect head movement function in response to speech presence / absence information indicating the presence or absence of speech.

[0044] The feature amount calculation unit 33 calculates the feature amount of the head movement function based on the detection result by the head movement function detection unit 32b. The feature amount calculation unit 33 also calculates the head kinematic feature amount based on the presence or absence of head movement detected by the head movement detection unit 32a. Furthermore, the feature amount calculation unit 33 calculates the head kinematic feature amount based on the presence or absence of speech detected by the head movement detection unit 32a.

[0045] The subjective impression prediction unit 34 predicts the subjective impression of the interlocutor for at least the head movement function feature quantity out of the head movement function feature quantity and the head kinematic feature quantity using a regression model. An example of output information (data) of the subjective impression prediction unit 34 is shown in FIG. 11 described later.

[0046] <Relationship between input and output information> Here, the relationship between the input information (data) and the output information (data) in each function will be described with reference to Fig. 4. Fig. 4 is a diagram showing the relationship between the input information (data) and the output information (data) in each function.

[0047] 4, when data acquisition unit 31 acquires video data, head movement detection unit 32a outputs information on the presence or absence of head movement if the input information is coordinate information related to head movement. Also, when data acquisition unit 31 acquires voice data or the like, head movement detection unit 32a outputs information on the presence or absence of speech.

[0048] Next, in head movement function detection section 32b, input information is at least one of information on the presence or absence of head movement and information on the presence or absence of speech, and output information is the detection result of head movement function.

[0049] Next, in the feature calculation unit 33, when the input information is the detection result of head movement function, the output information is the feature of head movement function (see FIG. 10 described later). In addition, in the feature calculation unit 33, when the input information is at least one of information on the presence or absence of head movement and information on the presence or absence of speech, the output information is head kinematics feature (see FIG. 9 described later). Note that while the detection result of head movement function is essential as input information, information on the presence or absence of head movement and information on the presence or absence of speech are not essential. In other words, the feature calculation unit 33 must calculate the feature of head movement function, but does not must calculate the feature of head kinematics.

[0050] Next, in subjective impression prediction section 34, input information is at least the head movement function feature amount out of the head movement function feature amount and the head kinematic feature amount, and output information is a predicted value of the subjective impression of the interlocutor.

[0051] [Processing or Operation of the Embodiment] Next, the processing or operation of this embodiment will be described with reference to FIGS.

[0052] <Data learning behavior> First, the behavior of data learning will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the behavior of data learning.

[0053] The data acquiring unit 31 receives interlocutor data output in time series from the input device 2 (S11). Then, the data acquiring unit 31 acquires coordinate information relating to head movement (information indicating a speech section) from the interlocutor data (S12).

[0054] Here, the processing of the data acquisition unit 31 will be specifically described.

[0055] (Use of video data) First, a case where the data acquisition unit 31 utilizes video data will be described. The data acquisition unit 31 acquires video data from the camera 2a, smartphone 2b, etc. When the data acquisition unit 31 receives the video data, it performs face tracking using OpenFace face recognition technology and measures the head attitude angle. The data acquisition unit 31 detects, for example, three components of the head attitude angle: Azimuth, Elevation, and Roll.

[0056] Note that, instead of the Azimuth, Elevation, and Roll angles, angles based on other coordinate systems may be used. Furthermore, the data acquisition unit 31 may detect angular acceleration.

[0057] The data acquisition unit 31 may acquire coordinate information related to head movement from any program generated by a computer based on the movement of a person in a dialogue video. In this case, the data acquisition unit 31 extracts coordinates output from the program corresponding to time frame t. Furthermore, the data acquisition unit 31 may acquire coordinate information related to head movement using a wearable sensor, an acceleration sensor, a motion capture, a magnetic sensor, or a device having these built-in. In this case, the data acquisition unit 31 extracts information on angular velocity and angular acceleration corresponding to time frame t.

[0058] As a result of the above, data acquisition section 31 sets each component in time frame t as the following values ​​as coordinate information related to head movement, and outputs these to head movement detection section 32a.

[0059]

number

[0060] When the data acquiring unit 31 acquires voice data as interlocutor data, the data acquiring unit 31 acquires speech section data from sound pressure information. For example, the data acquiring unit 31 detects a sound pressure having a value equal to or greater than an arbitrary threshold as a speaking section.

[0061] When the data acquisition unit 31 acquires video data as interlocutor data, it extracts voice data from the video data. The data acquisition unit 31 may also use the video data to extract certain feature points (for example, feature points around the mouth) that can be acquired by OpenFace face recognition technology, and may determine that the person is speaking if the motion vector of the feature points is equal to or greater than a certain threshold. In addition, when there is already a corpus in which the speech content for the video data is transcribed with time series information, the data acquisition unit 31 may use information from the corpus to determine the section in which the speech content exists as speech section data.

[0062] Furthermore, the data acquisition unit 31 may acquire the information from an arbitrary program generated by a computer based on the information of the speech section in the speech video. In this case, the head movement detection unit 32a described later extracts the information on the presence or absence of speech output from the program corresponding to the time frame t.

[0063] As a result, data acquiring unit 31 outputs information about the speech section (including information about time frame t) to head movement detecting unit 32a. As described above, data acquiring unit 31 may acquire data about head movement instead of coordinate information about head movement.

[0064] Next, head movement detection unit 32a performs machine learning using the head movement detection model to detect the presence or absence of head movement (presence or absence of speech) with respect to coordinate information related to head movement (information indicating a speech section) (S13). Note that, if data related to head movement is acquired in step S12, head movement detection unit 32a performs machine learning using the head movement detection model to detect the presence or absence of head movement with respect to the data related to head movement. Also, as described above, step S13 by head movement detection unit 32a may be omitted. In this case, the data acquired in step S12 is directly input to head movement function detection unit 32b.

[0065] Next, head movement detection section 32a stores the head movement detection model after machine learning in model storage section 39 (S14).

[0066] Next, head movement function detection unit 32b performs machine learning using the head movement function detection model to detect the head movement function for head movement presence / absence information (information on the presence / absence of speech) (S15). Note that if step S13 by head movement detection unit 32a is omitted, head movement function detection unit 32b performs machine learning using the head movement function detection model to detect the head movement function for the data acquired in step S12.

[0067] Next, head motor function detection section 32b stores the machine-learned head motor function detection model in model storage section 39 (S16).

[0068] This concludes the explanation of the behavior of data learning.

[0069] <Data prediction behavior> Next, the behavior of data prediction will be described with reference to Fig. 6 to Fig. 11. Fig. 6 is a flowchart showing the behavior of data prediction. Note that the following steps S21, S22, S23, and S24 are the same processes as the above-mentioned steps S11, S12, S13, and S15, respectively, and therefore descriptions of the overlapping processes will be omitted.

[0070] The data acquisition unit 31 receives interlocutor data output in time series from the input device 2 (S12). Then, the data acquisition unit 31 acquires coordinate information on head movement (information indicating a speech section) from the interlocutor data (S22). Note that the method of acquiring the coordinate information on head movement and the information indicating a speech section may be a method other than machine learning. Also, as described above, the data acquisition unit 31 may acquire data on head movement instead of coordinate information on head movement.

[0071] Next, head movement detection unit 32a uses the trained head movement detection model to detect the presence or absence of head movement (presence or absence of speech) with respect to coordinate information related to head movement (information indicating a speech section) (S23). Note that, if data related to head movement is acquired in step S22, head movement detection unit 32a uses the trained head movement detection model to detect the presence or absence of head movement with respect to the data related to head movement. Also, as described above, step S23 by head movement detection unit 32a may be omitted. In this case, the data acquired in step S22 is directly input to head movement function detection unit 32b.

[0072] Next, head movement function detection section 32b detects head movement function in response to head movement presence / absence information (information on the presence / absence of speech) using the trained head movement function detection model (S24).

[0073] In this case, head movement function detection unit 32b performs detection including at least one of the 10 types of head movement functions that have been machine-learned. When detecting all 10 types of head movement functions, head movement function detection unit 32b expresses the detection results of the 10 types of head movement functions in frame t during a conversation as follows, using the numbers shown in FIG. 8.

[0074]

number

[0075] As described above, if step S23 is omitted, head movement function detection section 32b uses the trained head movement function detection model to detect head movement function for the data acquired in step S22.

[0076] Next, feature amount calculation unit 33 calculates the feature amount of head movement function based on the detection result of the head movement function (S25). Also, feature amount calculation unit 33 calculates head kinematic feature amount based on the head movement presence / absence information (information on the presence / absence of speech) (S26). Note that this step S26 is not a required process.

[0077] Here, the process executed by the feature amount calculation unit 33 will be described in more detail.

[0078] The feature amount calculation unit 33 acquires at least the head movement function detection result from among the head movement presence / absence information and the head movement function detection result, and calculates the feature amount to be input to the regression model 50 at the next stage.

[0079] Below, we define six types of head kinematic features and 22 types of head movement function features, and explain the case where at least the 22 types of head movement function features are used. Note that the features used are not limited to these, and other features may also be used.

[0080] Fig. 9 is a diagram showing a list of kinematic features. Here, if the number of frames included in a slot is T, the total number of frames in which head movement is detected within the slot is NF, and the number of head movement intervals within the slot is NI, then each feature amount is defined as shown in Fig. 9. This will be explained in more detail.

[0081] Hrate indicates the ratio of frames in which head movement periods are detected to the total number of frames in one slot. Hnum indicates the number of head movement periods in each slot. Hdur indicates the average number of frames per head movement period detected in each slot. These are expressed by (Equation 1).

[0082]

number

[0083]

number

[0084]

number

[0085]

number

[0086]

number

[0087] (a) Function occurrence rate (first feature) Function appearance rate

[0088]

number

[0089]

number

[0090]

number

[0091] (b) Functional content (second feature) Functional content

[0092]

number

[0093]

number

[0094]

number

[0095]

number

[0096]

number

[0097] (c) Functional division composition ratio (third characteristic quantity) Functional category composition ratio

[0098]

number

[0099]

number

[0100]

number

[0101]

number

[0102] Also, if you are mostly active as a listener,

[0103]

number

[0104] The feature amount calculation section 33 outputs the feature amounts calculated as described above to the subjective impression prediction section .

[0105] Next, the subjective impression prediction unit 34 uses a regression model to output predicted values ​​(predicted subjective impressions) of each item of the interlocutor's introspection report score for at least the head movement function features out of the head movement function features and the head kinematic features (S27).

[0106] Here, the processing of the subjective impression prediction unit 34 will be described in more detail. FIG. 11 is a diagram showing predicted values ​​of subjective impressions as introspection report scores for each item. In FIG. 11, the four introspection report score items, "atmosphere," "fun," "motivation," and "concentration," are divided into two-minute intervals and values ​​are expressed as integers from 1 to 9. The integer values ​​may be an arbitrarily input value (A) or a value output from some device (B). Moreover, the integer values ​​may be values ​​obtained as a result of calculation of the values ​​(A) and (B) by an arbitrary method.

[0107] The subjective impression prediction unit 34 inputs the feature quantities obtained from each slot during the conversation into the machine-learned regression model 50, and outputs the predicted values ​​of each item of the introspection report score. The predicted values ​​may be either nine-level discrete values ​​or real values.

[0108] This concludes the explanation of the behavior of data prediction.

[0109] [Effects of the embodiment] As described above, the present embodiment has an effect of being able to predict the subjective impressions (feelings) of the interlocutors during a conversation. This leads to the establishment of technology that improves the conversation experience of the interlocutors, such as building relationships between the interlocutors and ensuring psychological safety.

[0110] In addition, by using both head movement during dialogue and features of head movement function, it becomes possible to make predictions that are robust to differences in situations during dialogue.

[0111] Furthermore, by taking into account the rate of occurrence of functions, it is possible to use the quantitative aspect of dialogue activity, by taking into account the rate of content of functions, it is possible to use the nature of speech activity, and by taking into account the composition ratio of functional categories, it is possible to use the outline of each participant's dialogue activity as a feature, thereby improving the accuracy of prediction.

[0112] 〔supplement〕 The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations). (1) The prediction device 3 of the present invention can be realized by a computer and a program. This program can be recorded on a recording medium or provided via a communication network. (2) In the above embodiment, a camera 2a or the like is shown as an example of the input device 2, but this is not limited to this and may be, for example, a desktop personal computer, a notebook computer, a tablet terminal, a smart watch, a car navigation device, etc. (3) Each of the CPUs 301 and 501 may be multiple, rather than just single. [Explanation of symbols]

[0113] 1. Communication Systems 3. Prediction Device 4 Monitors 31 Data acquisition unit (an example of a data acquisition means) 32 Detection unit 32a Head movement detection unit (an example of a head movement detection means) 32b Head movement function detection unit (an example of a head movement function detection means) 33 Feature calculation unit (an example of feature calculation means) 34 Subjective impression prediction unit (an example of a subjective impression prediction means) 39 Model storage unit (an example of model storage means) 40a Head movement detection model 40b Head movement function detection model 50 Regression Models

Claims

1. A prediction device for predicting a subjective impression held by a person in a conversation, comprising: A data acquisition means for acquiring data on head movement of the interlocutor from video data obtained by photographing the interlocutor; a head movement function detection means for detecting a head movement function for the data relating to the head movement using a head movement function detection model; a feature amount calculation means for calculating a feature amount of the head movement function based on a detection result by the head movement function detection means; a subjective impression prediction means for predicting a subjective impression of the interlocutor with respect to the head movement function feature quantity by using a regression model; A prediction device comprising:

2. A prediction device for predicting a subjective impression held by a person in a conversation, comprising: A data acquisition means for acquiring data on head movement of the interlocutor from video data obtained by photographing the interlocutor; head movement detection means for detecting the presence or absence of head movement with respect to the head movement related data using a head movement detection model; a head movement function detection means for detecting a head movement function corresponding to the head movement presence / absence information indicating the presence or absence of head movement using a head movement function detection model; a feature amount calculation means for calculating a feature amount of the head movement function based on a detection result by the head movement function detection means; a subjective impression prediction means for predicting a subjective impression of the interlocutor with respect to the head movement function feature quantity by using a regression model; A prediction device comprising:

3. the data regarding head movement is coordinate information regarding the head movement, 3. The prediction device according to claim 2, wherein the head movement detection means detects the presence or absence of the head movement with respect to the coordinate information relating to the head movement by using the head movement detection model.

4. the feature amount calculation means calculates a head kinematic feature amount based on the presence or absence of the head movement detected by the head movement detection means; The subjective impression prediction means predicts the subjective impression of the interlocutor with respect to the head movement function feature amount and the head kinematic feature amount by using the regression model. The prediction device according to claim 2 or 3,

5. A prediction device for predicting a subjective impression held by a person in a dialogue, comprising: a data acquisition means for receiving interlocutor data relating to an interlocutor and acquiring, from the interlocutor data, information on a speech section indicating an utterance section of the interlocutor; a head movement detection means for detecting the presence or absence of speech in the speech section information by using a head movement detection model; a head movement function detection means for detecting head movement function in response to the speech presence / absence information indicating the presence or absence of speech using a head movement function detection model; a feature amount calculation means for calculating a feature amount of the head movement function based on a detection result by the head movement function detection means; a subjective impression prediction means for predicting a subjective impression of the interlocutor with respect to the head movement function feature quantity by using a regression model; A prediction device comprising:

6. The feature amount calculation means calculates a head kinematic feature amount based on the presence or absence of the speech detected by the head movement detection means, The subjective impression prediction means predicts the subjective impression of the interlocutor with respect to the head movement function feature amount and the head kinematic feature amount by using the regression model. The prediction device according to claim 5 .

7. The prediction device according to claim 5 or 6, characterized in that the data acquisition means acquires information about the speech section from audio data included in the video data as the interlocutor data obtained by filming the interlocutor, acquires information about the speech section from audio data as the interlocutor data obtained by collecting the voice of the interlocutor, acquires information about the speech section by determining that the interlocutor has spoken when a motion vector of a feature point around the mouth of the interlocutor in the video data is equal to or greater than a predetermined threshold, or acquires information about the speech section from a corpus in which speech content for the video data is transcribed while having time series information.

8. A prediction method implemented by a prediction device for predicting a subjective impression held by a person in a dialogue, comprising: a data acquisition step of acquiring data on head movement of the interlocutor from video data obtained by photographing the interlocutor; a head movement function detection step of detecting a head movement function for the data relating to the head movement using a head movement function detection model; a feature amount calculation step of calculating a feature amount of the head movement function based on a detection result in the head movement function detection step; a subjective impression prediction step of predicting a subjective impression of the interlocutor with respect to the head movement function feature quantity using a regression model; A prediction method comprising the steps of:

9. A prediction method implemented by a prediction device for predicting a subjective impression held by a person in a dialogue, comprising: a data acquisition step of acquiring data on head movement of the interlocutor from video data obtained by photographing the interlocutor; a head movement detection step of detecting the presence or absence of head movement with respect to the head movement related data using a head movement detection model; a head movement function detection step of detecting a head movement function corresponding to head movement presence / absence information indicating the presence / absence of head movement using a head movement function detection model; a feature amount calculation step of calculating a feature amount of the head movement function based on a detection result by the head movement function detection step; a subjective impression prediction step of predicting a subjective impression of the interlocutor with respect to the head movement function feature quantity using a regression model; A prediction method comprising the steps of:

10. A prediction method executed by a prediction device for predicting a subjective impression held by a person in a dialogue, comprising: a data acquiring step of receiving interlocutor data related to an interlocutor and acquiring, from the interlocutor data, information on a speech section indicating an utterance section of the interlocutor; a head movement detection step of detecting the presence or absence of speech in the speech section information by using a head movement detection model; a head movement function detection step of detecting head movement function in response to the speech presence / absence information indicating the presence or absence of speech using a head movement function detection model; a feature amount calculation step of calculating a feature amount of the head movement function based on a detection result by the head movement function detection step; a subjective impression prediction step of predicting a subjective impression of the interlocutor with respect to the head movement function feature quantity using a regression model; A prediction method comprising the steps of:

11. A program causing a computer to execute the method according to any one of claims 8 to 10.

Images