Video communication device, and control device and control method used therefor

Abstract

To provide a video call device allowing a user to enjoy conversations visually in an online communication, such as TV phone.SOLUTION: A control apparatus for controlling a video call device having an imaging unit for acquiring video information and a microphone array composed of a plurality of microphones for acquiring voice includes: a person position specifying unit which detects a position of a person from the video information and voice sound collected by the microphone array; a person characteristic detection unit which detects characteristics of a person from the video information; a voice recognition unit which converts the voice sound collected by the microphone array to a text; a balloon image generation unit which generates a balloon image for displaying the text converted by the voice recognition unit; and a synthesizing display unit which synthesizes the balloon image near an image of a person who speaks. The balloon image or a font of the text to be displayed in the balloon image is modified by the balloon image generation unit in accordance with the characteristics of the person detected in the person characteristic detection unit.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to a video communication device for videoconferencing, videophones, and the like. [Background technology]

[0002] In recent years, the global spread of the novel coronavirus (COVID-19) has significantly changed the way people communicate. In order to avoid contact with people as much as possible, companies are using web conferencing and video conferencing for meetings, and individuals are also using online methods such as video calls on smartphones to communicate with relatives and friends in distant locations.

[0003] Video communication devices such as video conferencing and videophones use televisions, personal computers, smartphones, etc. to facilitate meetings and communication by capturing images of users in remote locations with a camera via the Internet and displaying the images on a display, while also picking up conversations with a microphone and broadcasting them to each user via a speaker.

[0004] In such video communication devices, when there are multiple participants, there are times when it is difficult to understand who is saying what. Also, participants with hearing impairments may have difficulty hearing the sound coming from the speaker.

[0005] A prior art document in this technical field is Patent Document 1. Patent Document 1 describes a method in which a speaker is identified by an individual directional microphone placed near the speaker, the spoken content is converted into text, and then the spoken content is synthesized and displayed as a speech bubble near the displayed person. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2009-194857 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, Patent Document 1 describes only a standardized example of a speech bubble image to be synthesized and displayed, and while this is fine for video conferences used for business purposes, it lacks appeal as a tool for personal video calls or casual occasions such as online drinking parties, which have become more common in recent years. In addition, because the speaker is identified by an individual directional microphone, if the speaker moves, they may be out of range of the directional microphone, and the spoken sound may not be captured correctly.

[0008] The present invention has been made in consideration of the above problems, and an object of the present invention is to provide a video communication device that allows users to visually enjoy conversations in online communication such as videophone calls. [Means for solving the problem]

[0009] As an example, the present invention is a control device that controls a video calling device having an imaging unit that acquires video information and a microphone array consisting of multiple microphones that acquire audio, and is configured to include a person position identification unit that detects a person's position from the video information and spoken sounds picked up by the microphone array, a person characteristic detection unit that detects the person's characteristics from the video information, a voice recognition unit that converts the spoken sounds picked up by the microphone array into text, a speech bubble image generation unit that generates a speech bubble image that displays the text converted by the voice recognition unit, and a synthesis display unit that synthesizes the speech bubble image near the image of the person who spoke, and is configured to change the font of the speech bubble image or the text to be displayed in the speech bubble image in the speech bubble image generation unit based on the results of detection of the person's characteristics by the person characteristic detection unit. Effect of the Invention

[0010] According to the present invention, it is possible to provide a video communication device that allows users to visually enjoy conversations in online communication such as videophone calls. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of an entire system including a video communication device according to a first embodiment. [Diagram 2] FIG. 2 is a functional block diagram of a control device according to the first embodiment. [Diagram 3] 4 is a process flowchart of the control device in the first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of a display image of the video calling device in the first embodiment. [Diagram 5] FIG. 2 is a diagram showing an example of a display image of the video calling device in the first embodiment. [Figure 6A] 3A to 3C are diagrams for explaining the principle of human position detection using a microphone in the first embodiment. [Figure 6B] 13 is another diagram for explaining the principle of human position detection using a microphone in the first embodiment. FIG. [Figure 7] FIG. 2 is a diagram showing an example of a display image of the video calling device in the first embodiment. [Figure 8] FIG. 2 is a diagram showing an example of a display image of the video calling device in the first embodiment. [Figure 9] FIG. 2 is a diagram showing an example of a display image of the video calling device in the first embodiment. [Figure 10] FIG. 2 is a diagram showing an example of a display image of the video calling device in the first embodiment. [Figure 11A] FIG. 11 is a diagram showing an example of a display image of the video calling device in the second embodiment. [Figure 11B] FIG. 11 is a diagram showing an example of a display image of the video calling device in the second embodiment. [Figure 12] FIG. 11 is a diagram showing an example of a display image of the video calling device in the third embodiment. [Figure 13A] 13 is a diagram for explaining the principle of human position detection using a microphone in the third embodiment. FIG. [Figure 13B] 13 is another diagram for explaining the principle of human position detection using a microphone in the third embodiment. FIG. [Figure 13C] 13 is another diagram for explaining the principle of human position detection using a microphone in the third embodiment. FIG. [Figure 14] FIG. 11 is a diagram showing an example of a display image of the video calling device in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. EXAMPLES

[0013] FIG. 1 is a block diagram showing the overall configuration of a system including a video communication device according to this embodiment.

[0014] In Fig. 1, a video communication device 1 is composed of a control device 10, a display 11, a camera 12, a microphone 13, and a speaker 14. There are multiple video communication devices 1 for each user, for example, there are three video communication devices 1 in Fig. 1, and each video communication device 1 is connected to an information communication network 2 such as the Internet or an intranet. The video communication devices 1 and the information communication network 2 are connected via a wired LAN (Local Area Network) or a wireless LAN (not shown).

[0015] In actual implementation of the video communication device 1, for example, in the case of a notebook computer or a smartphone, the control device 10, display 11, camera 12, microphone 13, and speaker 14 are often all built in. In the case of a desktop computer, the main body is the control device 10, and the other components such as the display 11, camera 12, microphone 13, and speaker 14 can be prepared separately. The control device 10 can be an inexpensive SBC (Single Board Computer) or a dedicated video conference terminal device other than a desktop computer.

[0016] The operation of the video communication device 1 will be described. The control device 10 is composed of a CPU (Central Processing Unit) (not shown), a volatile memory such as an SDRAM (Synchronous Dynamic Random Access Memory), a hard disk (HDD), a solid state drive (SSD), a non-volatile storage medium such as a flash memory, etc. In a personal computer, it corresponds to a motherboard. The camera 12 is an imaging unit that captures a user in the vicinity of the video communication device 1, and outputs the captured image information to the control device 10. The type of the camera 12 may be a normal two-dimensional camera or a three-dimensional camera that can measure the distance of the captured object. The microphone 13 is a voice acquisition unit that captures the voice uttered by a user in the vicinity of the video communication device 1, and outputs the acquired voice data to the control device 10. The type of the microphone 13 includes a large dynamic microphone, a condenser microphone, a small MEMS microphone, etc., and a microphone array consisting of two or more microphones is used to detect the position of the user, as will be described later. The display 11 displays the video data sent from the other video communication device 1 via the information and communication network 2, and can also display a close-up image of the user himself / herself in real time as necessary. The speaker 14 reproduces the audio data sent from the other video communication device 1 via the information and communication network 2.

[0017] Next, the operation of the video communication device 1 will be described in detail with reference to Figures 2 to 10. Figure 2 is a functional block diagram showing the functions realized by the control device 10. The hardware is composed of the above-mentioned CPU, volatile memory, non-volatile storage medium, etc. (not shown), and the CPU executes a program stored in the non-volatile storage medium while expanding it into the volatile memory. Figure 3 is a processing flowchart of the control device 10. That is, the processing of the control device 10 is executed by a program stored in the non-volatile storage medium.

[0018] The operation of the video calling device 1 will be described below according to the flowchart in Fig. 3, with reference to Fig. 2 as well. For example, assume that there are two video calling devices 1 in the system, with three family members on one device and grandparents on the other device making a video call.

[0019] In Fig. 3, S101 is the state in which each video communication device 1 is powered on and video data of users in the vicinity of each video communication device 1 is displayed on the display 11 via the information communication network 2. That is, a close-up of a family of three parents and children is displayed on the display 11 on the grandparents' side, as shown in Fig. 4. At this stage, the three parents and children have not yet spoken anything.

[0020] S102 is a "person position detection (camera)" process. In FIG. 2, the person position identification unit 101 detects the faces of people from the video captured by the camera 12 on the parent-child family side as shown in FIG. 5, and assigns individual ID attributes. The person's face detection is performed by AI processing by the person position identification unit 101. Since the person's face detection by AI processing is a known technology, the processing content will be omitted. In the example of FIG. 5, the father is assigned ID1, the child is assigned ID2, and the mother is assigned ID3. In addition, the position of the ID on the screen is attached to the ID attribute. For example, the position is angle information from the center of the screen, with the center of the screen being 0°, the left side being negative, and the right side being positive. For example, -30° is attached to the father's ID1, 0° to the child's ID2, and +30° to the mother's ID3. The calculation of the angle information by the camera 12 can be uniquely calculated if the lens angle of view (field of view) of the camera 12 is determined. If the left and right edges of the image captured by camera 12 are a maximum of ±60° as shown in Figure 2, then the position between ±60° can be linearly determined depending on the position on the image. For example, the center of the screen can be calculated as 0°, and if it is exactly halfway between the center and the left and right edges, it can be calculated as ±30°.

[0021] S103 is the "presence or absence of speech" determination process. In Fig. 2, the voice acquisition and generation unit 103 determines whether or not there is speech collected by the microphone 13 on the parent-child family side, and if there is speech, the process proceeds to S104, and if there is no speech, the process returns to S102. If there is speech, the voice data is sent to the voice acquisition and generation unit 103 on the grandparents' side via the information and communication network 2, and the speech is reproduced on the speaker 14 on the grandparents' side.

[0022] S104 is a "voice position detection (microphone)" process. From the voice data collected by the microphone 13 on the parent-child family side, the person position identification unit 101 identifies the position where the voice data was spoken. If there are two or more microphones to identify the position, the angle to each microphone can be calculated from the delay time of the voice reaching each microphone, and from that angle, it becomes possible to calculate relative angle information between the video communication device 1 and the user who spoke.

[0023] FIG. 6A shows a specific example of a method for calculating the angle of a user using two microphones. Take father ID1 speaking from the left side of the screen as an example. In FIG. 6A, two microphones, microphones 13a and 13b, are placed at a distance d0 (m) on a line L1 at a predetermined distance. When father ID1 speaks from the left side of the screen, assume that there is a delay of Δt (sec) in the voice acquisition start time of microphone 13b compared to microphone 13a. The delay of Δt (sec) is caused by the difference in distance from the speaking point to microphones 13a and 13b, so the distance difference d1 can be calculated by sound speed × Δt. It is known that the sound speed at an air temperature of 20°C is 343.5 (m / sec). For example, if Δt is 145.56e-6 (sec), it can be calculated as d1 = 343.5 × 145.56e-6 = 0.05 (m).

[0024] The angle α1 in Figure 6A can be approximated as 90 degrees if we consider that the distance between father ID1 and microphone 13b is sufficiently longer than the distance between microphones 13a and 13b. Then, we can approximate α' = α, and we can calculate the angle α = arc SIN (d1 / d0). If the distance d0 between the two microphones is 0.1 (m), then the angle α = arc SIN (0.05 / 0.1) = 30°, and since father ID1 is on the left side, we can calculate it as -30°.

[0025] FIG. 6B shows another example of a method for calculating the angle of a user using two microphones. In FIG. 6B, microphones 13a and 13b are arranged on a line L1 at a predetermined distance and on a line L2 at a distance d2 from line L1. On line L1, the positions of microphones 13a and 13b projected on line L1 are separated by a distance d0 (m). When father ID1 speaks from the left side of the screen, it is assumed that there is a delay of Δt (sec) in the voice acquisition start time of microphone 13a compared to microphone 13b. The delay of Δt (sec) is caused by the difference in distance between microphones 13b and 13a from the point of speech, so the distance difference d1 can be calculated by the speed of sound × Δt. If angle α1 in FIG. 6B is approximated to 90 degrees, angle α2 = arc SIN (d1 / d3) can be calculated. d3 can be calculated from d0 and d2, and α3 can also be calculated, and therefore angle α can also be calculated.

[0026] Returning to FIG. 3, S105 is a "camera & audio position match" determination process. Here, it is determined whether the angle associated with the person ID acquired in S102 matches the audio angle calculated in S104, and if they match, the process proceeds to S106, and if they do not match, the process returns to S102. An example of a mismatch would be when the voice of a person not within the shooting screen of the camera 12 or surrounding sounds are acquired. Also, when there are multiple people within the shooting screen as in this embodiment, the subsequent processes are performed only on the person IDs whose angles of the camera 12 and audio match.

[0027] S106 is a "speech to text conversion" process. The speech data whose angle matches the person ID in S105 is converted to text by the speech recognition unit 104 on the parent-child family side. Since speech to text conversion technology is a well-known technology, the details of the process will be omitted here. Speech to text conversion can be performed by the speech recognition unit 104 shown in Fig. 2, but if there are problems with the limited number of registered words or processing speed, a speech recognition cloud service on the Internet may be used.

[0028] Figure 7 shows an example of speech-to-text conversion. Father ID1 pronounces "It's been a long time since we last saw each other, Mom." This is converted to text "It's been six months since we last saw each other, Mom." Child ID2 pronounces "Grandma, how are you?" This is converted to text "Grandma, how are you?" Mother ID3 pronounces "It's been a long time, Mom." This is converted to text "It's been a long time, Mom."

[0029] S107 is the "gender, age, and emotion detection" process. For each person ID assigned in S102, the gender, age, and emotion are determined using AI processing by the parent-child family side person characteristic emotion detection unit 102 shown in Figure 2, and the attributes are recorded for each ID. Using Figure 5 as an example, ID1 is detected as male, in his 40s, and emotion of joy. ID2 is detected as male, in his teens, and emotion of joy. ID3 is detected as female, in his 40s, and emotion of joy. Since the detection of a person's gender, age, and emotion using AI processing is a publicly known technology, the details of the process are omitted.

[0030] S108 is a "speech bubble selection" process. Based on the gender, age, and emotion information for each ID set in S107, the text balloon to be displayed is changed by the parent-child family balloon and font generation unit 105 shown in FIG.

[0031] Figure 7 is an example of a speech bubble before the change, which is a featureless square speech bubble image. If there is no information on "gender, age, emotion detection", for example, this square speech bubble image is used. Figure 8 is an example of a speech bubble after the change, which can be set to, for example, a square speech bubble image with rounded corners for a man in his 40s, an entirely round speech bubble image for a woman in her 40s, and a cloud-like speech bubble image for a child. It is also possible to change the speech bubble according to the emotion of each ID, as shown in Figure 9. The right is an example of a speech bubble for "happy", and the left is an example of a speech bubble for "sad". These settings can be selected by the user from multiple speech bubble images in advance.

[0032] S109 is a "font selection" process. It changes the font of the text displayed by the speech bubble font generation unit 105 on the parent-child family side shown in Fig. 2. Fig. 7 is an example of a font before the change, for example Mincho. Fig. 8 is an example of a font after the change, for example Gothic for those in their 40s and Pop for children. This setting can be selected in advance from multiple fonts by the user.

[0033] S110 is the "emoji insertion unit" process. Emojis are inserted into the text by the speech bubble and font generation unit 105 on the parent-child family side shown in Figure 2. Emojis corresponding to, for example, "happy" and "sad" are registered in advance based on emotion information for each ID, and are inserted at the end of the text. The user can also set in advance whether or not to insert them. Figure 9 shows an example of an emoji 115 for "happy" on the right, and an example of an emoji 116 for "sad" on the left.

[0034] S111 is the "person speech bubble synthesis unit" process. The speech bubble image, text content, font, and pictogram information selected in S106 to S110 are transmitted to the grandparent's video calling device 1 via the information and communication network 2, and the synthesis and display unit 106 on the grandparent's side synthesizes and displays the images on the grandparent's display 11 adjacent to the ID images of the parent and child. If necessary, the synthesis and display unit 106 on the parent-child family side shown in Fig. 2 may also synthesize and display the images adjacent to the ID images in the same manner, as shown in Fig. 10, and further synthesize and display the images on the parent-child family display 11 in picture-in-picture format.

[0035] After the processes from S102 to S111 are completed, the process returns to S102 and the series of processes are repeated. The composite video processed from S102 to S111 is transmitted to the other video communication device 1 via the information communication network 2.

[0036] The user can set in advance whether the speech bubble image will be displayed or erased after the speech is finished. The user can also set in advance whether the next speech will be added to the previous text or erased. Furthermore, if the next speech is added to the previous text, the user can set in advance how much text will be left.

[0037] Furthermore, in this embodiment, it is assumed that there are multiple video calling devices 1 connected to the information and communication network 2, but this is not limiting, and it is also possible that there is only one video calling device 1, the image captured by the camera 12 is displayed on the display 11 of the device, and spoken information is synthesized as a balloon image. For example, the configuration of this embodiment is applied to an unmanned ordering and payment terminal at a fast food restaurant, and the user can order products and select the payment method by voice, and the information is displayed as a balloon image, allowing the user to confirm from the balloon image whether the order and payment method are as intended.

[0038] Also, a person's attributes (male, female, etc.) may be determined from the frequency of the person's voice picked up by a microphone, and this may be used as a supplement to the attribute determination by the camera.

[0039] As described above, according to this embodiment, in online communication such as video conferences and videophones for personal use, a video communication device can be provided that can be used as a tool in casual situations to liven up the atmosphere and visually enjoy conversations by changing the font of speech bubbles and texts to be compositely displayed according to the characteristics and emotions of each individual to be displayed. Also, for example, a videophone can be used for online homecoming visits to allow hearing-impaired grandparents to visually enjoy conversations with their grandchildren. EXAMPLES

[0040] In this embodiment, a process will be described in which a user moves during online communication such as a videophone call.

[0041] 11A and 11B are diagrams showing a specific example of this embodiment. The video communication device 1 of this embodiment is the same as that of the first embodiment shown in FIG. 2. The flowchart of the process to be executed is also the same as that of the first embodiment shown in FIG.

[0042] In Fig. 11A, the father is assigned the ID attribute of ID1, the child is assigned the ID attribute of ID2, and the mother is assigned the ID attribute of ID3, and speech bubble images of each of the three people are synthesized near the people in the same situation as Fig. 8 in the first embodiment. Here, let us assume that the father ID1 moves to the right, and the child ID2 and the mother ID3 move to the left, as shown in Fig. 11B.

[0043] In the "person speech bubble synthesis unit" process in S111 of the flowchart in Fig. 3, when a person with each ID moves, the synthesis / display unit 106 synthesizes a speech bubble image so that the image is always close to the person with that ID. As a result, as shown in Fig. 11B, the speech bubble image of father ID1 can move to the right in the same way as father ID1, and the speech bubble images of child ID2 and mother ID3 can also move to the left.

[0044] In addition, the user can set in advance whether or not to have the speech bubble image follow the user's movement. If you do not want the speech bubble image to follow the user's movement, you can simply erase the speech bubble image when the movement of a person with each ID is detected.

[0045] In this embodiment, the balloon image may or may not be changed according to the person's characteristics as in the first embodiment.

[0046] As described above, according to this embodiment, in online communications such as videophone, even if a user moves, the speech bubble images follow the user, which is expected to have the effect of making it easier to visually view the user's past comments. EXAMPLES

[0047] In this embodiment, a process for a case where a plurality of users overlap in the depth direction will be described.

[0048] As shown in FIG. 12, assume that the father ID1 is in the back and the child ID2 is in the front, overlapping at an angle α. In the configuration of the first embodiment, the person ID detection by the camera 12 is possible even if the users overlap if a three-dimensional camera is used for the camera 12. On the other hand, as for the voice, when two microphones 13a and 13b are arranged on a line L1 at a predetermined distance from the user as shown in FIG. 6A of the first embodiment, the voices of the father ID1 and the child ID2 are both from the angle α, and the voice results cannot be assigned to IDs. Also, as shown in FIG. 6B, even if the two microphones 13a and 13b are arranged at different positions in the front, back, left and right, if the father ID1 and the child ID2 are lined up on the line connecting the microphones 13a and 13b, the father ID1 and the child ID2 cannot be differentiated from each other, and the voice results cannot be assigned to IDs.

[0049] In order to solve this problem, in this embodiment, three microphones 13 are arranged. A specific example of a method for calculating the angle of a user using three microphones will be described below.

[0050] Fig. 13A shows a case where two of the three microphones 13a, 13b, and 13c, 13a and 13b, are not on an extension line connecting father ID1 and child ID2. In Fig. 13A, similar to Fig. 6B, microphones 13a and 13b are respectively placed on line L1 at a predetermined distance and on line L2 at a distance d2 from line L1. Furthermore, on line L1, the positions of microphones 13a and 13b projected onto line L1 are spaced a distance d0 apart.

[0051] At this time, due to the difference in distance between father ID1 and child ID2, a difference occurs between the distance difference d3 of the voice from father ID1 and the distance difference d4 of the voice from child ID2 in proportion to the difference in voice arrival time between microphone 13a and microphone 13b. Specifically, for example, with respect to the voice from child ID2, angle α1 in FIG. 13A can be approximated to 90 degrees, and angle α2 can be calculated as arc SIN(d4 / d5). d5 can be calculated from d0 and d2, and therefore angle β1 that identifies the position where the voice was generated from child ID2 can also be calculated. Similarly, angle β2 that identifies the position where the voice was generated from father ID1 can also be calculated.

[0052] Next, when the father ID1 and the child ID2 move and line up on the extension line connecting the two microphones 13a and 13b, the father ID1 and the child ID2 cannot be differentiated by the microphones 13a and 13b. In that case, as shown in FIG. 13B, by using the microphone 13c that is not on the extension line connecting the father ID1 and the child ID2, the father ID1 and the child ID2 can be differentiated. That is, in FIG. 13B, a difference occurs between the distance difference d3 of the voice from the father ID1 and the distance difference d4 of the voice from the child ID2 in proportion to the difference in voice arrival time between the microphones 13b and 13c. For example, for the voice from the child ID2, the angle α1 can be approximated to 90 degrees, and the angle β1 that identifies the position where the voice was generated from the child ID2 can be calculated as arc COS(d4 / d2). Similarly, the angle β2 that identifies the position where the voice was generated from the father ID1 can also be calculated.

[0053] FIG. 13C shows a case where three microphones 13a, 13b, and 13c are arranged on a line L1 at a predetermined distance. In FIG. 13C, for example, when microphones 13b and 13c are used, a difference occurs between the distance difference d3 of the voice from father ID1 and the distance difference d4 of the voice from child ID2 in proportion to the difference in voice arrival time between microphones 13b and 13c. For example, for the voice from child ID2, in FIG. 13C, angle α1 can be approximated to 90 degrees, and angle β1=arc SIN(d4 / d0) that identifies the position where the voice was generated from child ID2 can be calculated. Similarly, angle β2 that identifies the position where the voice was generated from father ID1 can also be calculated.

[0054] In this way, with a minimum of three microphones, it is possible to detect the geometric angles that identify the locations of multiple people speaking, regardless of their placement.

[0055] As described above, the difference between the detected angles β1 and β2 determines whether the vocalization results should be assigned to father ID1 or child ID2. In other words, the camera 12 can determine whether father ID1 or child ID2 is closer. On the other hand, depending on the arrangement of the microphones used, it can be determined whether angles β1 or β2 are geometrically closer in vocalization distance, so those vocalization results can be assigned to IDs. From these results, speech bubbles for father ID1 and child ID2 are synthesized based on the above associations, as shown in Figure 14.

[0056] As described above, according to this embodiment, even when a plurality of users overlap in the depth direction, it is expected that speech bubble images can be synthesized according to their positions.

[0057] Although the embodiments have been described above, the present invention is not limited to the above-mentioned embodiments, and various modified examples are included. For example, the above-mentioned embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. [Explanation of symbols]

[0058] 1: video communication device, 2: information and communication network, 10: control device, 11: display, 12: camera, 13: microphone, 14: speaker, 101: person position identification unit, 102: person characteristic emotion detection unit, 103: voice acquisition and generation unit, 104: voice recognition unit, 105: speech bubble and font generation unit, 106: synthesis and display unit, 115, 116: emoticons

Claims

1. A video communication device having an imaging unit for acquiring video information and an audio acquisition unit for acquiring audio, The voice capture unit is a microphone array including a plurality of microphones, a display that displays image information from the imaging unit or image information distributed from an external source via an information communication network; A control device is provided. The control device includes: a person position identification unit that, when the video information includes video information of a plurality of people, identifies the plurality of people by detecting human faces from the video information, assigns a person ID to each of the identified plurality of people, assigns an associated angle to the person ID from the video information that indicates a position on the screen of the person corresponding to the person ID, and detects a sound angle that indicates the position of the speech sound from the speech sound collected by the microphone array; a person characteristic detection unit that detects an emotion of a person corresponding to the person ID for each person ID from the video information; a voice recognition unit that converts into text only the voice collected by the microphone array and for which the voice angle that matches the associated angle associated with the person ID has been detected; a speech bubble image generating unit that generates a speech bubble image displaying text converted by the voice recognition unit for each person ID whose associated angle matches the voice angle, the speech bubble image generating unit performing at least one of changing a font of the text displayed in the speech bubble image and adding an emoticon to the text displayed in the speech bubble image according to a result of detection of the person's emotion by the person feature detection unit; a synthesis display unit that synthesizes the speech bubble image corresponding to the person ID near an image of a person corresponding to the person ID for each of the person IDs whose associated angle matches the audio angle; Equipped with Video communication device.

2. A video communication device having an imaging unit for acquiring video information and an audio acquisition unit for acquiring audio, The voice capture unit is a microphone array including a plurality of microphones, a display that displays image information from the imaging unit or image information distributed from an external source via an information communication network; A control device is provided. The control device includes: a person position identification unit that, when the video information includes video information of a plurality of people, identifies the plurality of people by detecting human faces from the video information, assigns a person ID to each of the identified plurality of people, assigns an associated angle to the person ID from the video information that indicates a position on the screen of the person corresponding to the person ID, and detects a sound angle that indicates the position of the speech sound from the speech sound collected by the microphone array; a person characteristic detection unit that detects a person characteristic including an emotion of a person corresponding to the person ID for each person ID from the video information; a voice recognition unit that converts into text only the voice collected by the microphone array and for which the voice angle that matches the associated angle associated with the person ID has been detected; a speech bubble image generating unit that generates a speech bubble image displaying text converted by the voice recognition unit for each of the person IDs whose associated angle matches the voice angle, the speech bubble image generating unit changing a font of the speech bubble image or the text displayed in the speech bubble image according to a result of detecting a person's characteristics by the person characteristics detecting unit, and adding an emoticon to the text according to a result of detecting a person's emotion by the person characteristics detecting unit; a synthesis display unit that synthesizes the speech bubble image corresponding to the person ID near an image of a person corresponding to the person ID for each of the person IDs whose associated angle matches the audio angle; Equipped with Video communication device.

3. 2. The video communication device according to claim 1, The control device includes: When the person position specifying unit detects a movement of a person's position, the synthesis display unit synthesizes the speech balloon image in the vicinity of the person who has moved.

4. A control device for controlling a video communication device having an imaging unit for acquiring video information and a microphone array including a plurality of microphones for acquiring audio, a person position identification unit that, when the video information includes video information of a plurality of people, identifies the plurality of people by detecting human faces from the video information, assigns a person ID to each of the identified plurality of people, assigns an associated angle to the person ID from the video information that indicates a position on the screen of the person corresponding to the person ID, and detects a sound angle that indicates the position of the speech sound from the speech sound collected by the microphone array; a person characteristic detection unit that detects an emotion of a person corresponding to the person ID for each person ID from the video information; a voice recognition unit that converts into text only the voice collected by the microphone array and for which the voice angle that matches the associated angle associated with the person ID has been detected; a speech bubble image generating unit that generates a speech bubble image displaying text converted by the voice recognition unit for each person ID whose associated angle matches the voice angle, the speech bubble image generating unit performing at least one of changing a font of the text displayed in the speech bubble image and adding an emoticon to the text displayed in the speech bubble image according to a result of detection of the person's emotion by the person feature detection unit; a synthesis display unit that synthesizes the speech bubble image corresponding to the person ID near an image of a person corresponding to the person ID for each of the person IDs whose associated angle matches the audio angle; Equipped with Control device.

5. A control device for controlling a video communication device having an imaging unit for acquiring video information and a microphone array including a plurality of microphones for acquiring audio, a person position identification unit that, when the video information includes video information of a plurality of people, identifies the plurality of people by detecting human faces from the video information, assigns a person ID to each of the identified plurality of people, assigns an associated angle to the person ID from the video information that indicates a position on the screen of the person corresponding to the person ID, and detects a sound angle that indicates the position of the speech sound from the speech sound collected by the microphone array; a person characteristic detection unit that detects a person characteristic including an emotion of a person corresponding to the person ID for each person ID from the video information; a voice recognition unit that converts into text only the voice collected by the microphone array and for which the voice angle that matches the associated angle associated with the person ID has been detected; a speech bubble image generating unit that generates a speech bubble image displaying text converted by the voice recognition unit for each of the person IDs whose associated angle matches the voice angle, the speech bubble image generating unit changing a font of the speech bubble image or the text displayed in the speech bubble image according to a result of detecting a person's characteristics by the person characteristics detecting unit, and adding an emoticon to the text according to a result of detecting a person's emotion by the person characteristics detecting unit; a synthesis display unit that synthesizes the speech bubble image corresponding to the person ID near an image of a person corresponding to the person ID for each of the person IDs whose associated angle matches the audio angle; Equipped with Control device.

6. The control device according to claim 4, When the person position identification unit detects a movement of a person's position, the synthesis display unit synthesizes the speech bubble image in the vicinity of the person who has moved.

7. A control method for controlling a video communication device having an imaging unit for acquiring video information and a microphone array including a plurality of microphones for acquiring audio, comprising: When the video information includes video information of a plurality of people, the plurality of people are identified by performing face detection from the video information, a person ID is assigned to each of the identified plurality of people, an associated angle indicating a position on the screen of a person corresponding to the person ID from the video information is assigned to the person ID, and a sound angle indicating the position of the speech sound is detected from the speech sound collected by the microphone array, Detecting an emotion of a person corresponding to each person ID from the video information, Converting into text only the vocalizations collected by the microphone array for which the voice angle that matches the associated angle associated with the person ID has been detected; generating a speech bubble image displaying the converted text for each person ID whose associated angle matches the voice angle; at least one of changing a font of text to be displayed in the speech bubble image and adding an emoji to the text to be displayed in the speech bubble image according to the detected human emotion; For each person ID in which the associated angle coincides with the audio angle, the speech bubble image corresponding to the person ID is synthesized in the vicinity of an image of a person corresponding to the person ID. Control methods.

8. A control method for controlling a video communication device having an imaging unit for acquiring video information and a microphone array including a plurality of microphones for acquiring audio, comprising: When the video information includes video information of a plurality of people, the plurality of people are identified by performing face detection from the video information, a person ID is assigned to each of the identified plurality of people, an associated angle indicating a position on the screen of a person corresponding to the person ID from the video information is assigned to the person ID, and a sound angle indicating the position of the speech sound is detected from the speech sound collected by the microphone array, Detecting a person's characteristics including an emotion of a person corresponding to the person ID from the video information for each person ID; Converting into text only the vocalizations collected by the microphone array for which the voice angle that matches the associated angle associated with the person ID has been detected; generating a speech bubble image displaying the converted text for each person ID whose associated angle matches the voice angle; synthesizing the speech bubble image in the vicinity of an image of the person who spoke; changing a font of the speech bubble image or a font of text to be displayed in the speech bubble image according to the detected person characteristic; adding emojis to the text in response to the detected human emotions; For each person ID in which the associated angle coincides with the audio angle, the speech bubble image corresponding to the person ID is synthesized in the vicinity of an image of a person corresponding to the person ID. Control methods.

9. 8. The control method according to claim 7, A control method comprising the steps of: synthesizing the speech bubble image in the vicinity of the person who has moved when a change in position of the person is detected.

Images