Video conference system
By converting audio into characters in real time in the video conferencing system and displaying it at the right time, the problem of users missing important information is solved, and information visibility and historical information search efficiency are improved.
Patent Information
- Application Number
- JP2023118504
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-07-20
- Publication Date
- 2025-01-30
Smart Images
Figure 2025015230000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to videoconferencing systems. [Background technology]
[0002] During video conferences or web conferences, there may be times when you cannot hear other users speaking during the conference, or you miss their audio.
[0003] As a method for solving such problems, a voice recognition system has been proposed that includes an area extraction means for extracting a specific area contained in first text data, which is a transcription of an utterance, and a text conversion means for converting the first text data into second text data based on the extracted specific area, and that uses the converted second text data for machine learning of the voice recognition engine (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2022-142374 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, in a conventional video conference system, if characters are constantly displayed on the screen, there is a risk that users will stop paying attention to the displayed characters. Furthermore, when a listener misses important or necessary information, there is a high possibility that the listener will also miss the corresponding text display.
[0006] Furthermore, in conventional video conference systems, if a recipient temporarily leaves the conference room, he or she cannot see the text display, so in this case too there is a risk that the recipient will miss important or necessary information.
[0007] On the other hand, conventional video conferencing systems allow users to check the entire history of text data after the conference has ended, but this takes time and effort to search through the entire history for important or necessary information that recipients may have missed.
[0008] This disclosure has been made in consideration of the above circumstances, and provides a video conference system that displays characters at more appropriate timing than conventional systems so that the recipient can easily focus on the characters. [Means for solving the problem]
[0009] The video conferencing system disclosed herein is a video conferencing system in which images and audio are shared via a plurality of user terminals connected via a network, the user terminals comprising an imaging unit for capturing an image of the user's face, an audio acquisition unit for capturing the user's voice, a communication unit for transmitting and receiving various data including image and audio data to and from other user terminals, a display unit for displaying the user's facial image on a video conferencing screen, an audio output unit for outputting the user's voice, an audio display unit for converting the user's voice into text and displaying it on the display unit, and a control unit for controlling the imaging unit, the audio acquisition unit, the communication unit, the display unit, the audio output unit and the audio display unit, and the control unit is characterized in that when predetermined audio display conditions are met during the video conferencing, the control unit causes the audio display unit to convert the user's voice being spoken into text and display it on the display unit. Effect of the Invention
[0010] According to this disclosure, when certain voice display conditions are met, the voice of a speaking user is converted into text and displayed, thereby enabling the realization of a videoconferencing system that displays text at a more appropriate timing than conventional systems so that the listener can easily focus on it. [Brief description of the drawings]
[0011] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of a configuration of a video conference system according to the present disclosure. [Diagram 2] 2 is a block diagram showing a schematic configuration of the user terminal of FIG. 1. [Diagram 3] FIG. 2 is a block diagram showing a schematic configuration of the server in FIG. 1. [Figure 4] 4 is a flowchart showing an example of a voice data display process of the video conference system of FIG. 1. [Diagram 5] 2 shows an example of a video conference screen displayed on a display unit of a user terminal in the video conference system of FIG. [Figure 6] 1 shows an example of a setting screen displayed when a setting button for the missed call function displayed on the display unit of a user terminal is pressed in the video conference system of FIG. [Figure 7] 2 shows an example of a setting screen for the missed broadcast function displayed on the display unit of a user terminal in the video conference system of FIG. 1. [Figure 8] 8 shows an example of an operation setting screen when the "Operation" button is selected on the setting screen for the missed program function in FIG. 7. [Figure 9] 8 shows an example of a person setting screen when the "People" button is selected on the setting screen for the missed program function in FIG. 7. [Figure 10] 8 shows an example of a word setting screen when the "Word" button is selected on the setting screen for the missed program function in FIG. 7. [Figure 11] 8 shows an example of an environment setting screen when the “Environment” button is selected on the setting screen for the missed program function in FIG. 7 . [Figure 12] 8 shows an example of a history setting screen when the "History" button is selected on the setting screen for the missed program function in FIG. 7. [Figure 13] 1 shows an example of a history confirmation display of missed conversation results on a video conference screen displayed on a display unit of a user terminal in the video conference system of FIG. [Figure 14] 1 shows an example of a history display of missed conversations on a video conference screen displayed on a display unit of a user terminal in the video conference system of FIG. [Figure 15]13 shows another example of a history display of missed conversations on a video conference screen displayed on a display unit of a user terminal in the video conference system of FIG. [Figure 16] 11 is a flowchart showing an example of a voice data display process of the video conference system according to the second embodiment of the present disclosure. [Figure 17] 13 shows an example of a video conference screen displayed on a display unit of a user terminal in the video conference system according to the second embodiment of the present disclosure. [Figure 18] 13 shows an example of a setting screen for a missed call function displayed on a display unit of a user terminal in the video conference system according to the third embodiment of the present disclosure. [Figure 19] 19 shows an example of an operation setting screen when the "Speak" button is selected on the setting screen for the missed broadcast function in FIG. [Figure 20] 13 shows an example of the display of audio data on a video conference screen displayed on a display unit of a user terminal in the video conference system according to the third embodiment of the present disclosure. [Figure 21] 13 shows an example of the display of audio data on a video conference screen displayed on a display unit of a user terminal when a plurality of speakers speak simultaneously in the video conference system according to the third embodiment of the present disclosure. [Figure 22] 13 shows an example of the display of audio data on a video conference screen displayed on a display unit of a user terminal when a speaker is wearing a mask in the video conference system according to the third embodiment of the present disclosure. [Diagram 23] 13 shows an example of the display of audio data on a video conference screen displayed on a display unit of a user terminal when a speaker is speaking while displaying a document in the video conference system according to the third embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Further, preferred embodiments of this disclosure will be described.
[0013] The imaging unit may image the user's movements, and the control unit may determine that the voice display condition is satisfied when it detects that the user, who is the listener, has performed a predetermined movement.
[0014] "When it is detected that the user who is the recipient has performed a predetermined action" includes, for example, when the recipient removes headphones / earphones, leaves their seat, or looks away for a predetermined period of time. In this case, the user terminal of the recipient converts all of the user's voice during speech into characters and displays them on a display unit.
[0015] In this way, the user's voice is converted into text and shown on the display at times when the recipient might miss what is being said, such as when the recipient removes their headphones or earphones, making it possible to realize a videoconferencing system that displays text at a more appropriate time than before to make it easier for the recipient to focus on.
[0016] The control unit may determine that the voice display condition is satisfied when detecting that a predetermined speech has been spoken by the user.
[0017] In this way, when a specific speaker speaks, the voice of that user is converted into text and displayed on the display unit, thereby realizing a videoconferencing system that displays text at a more appropriate time than before so that it is easier for the listener to focus on.
[0018] The control unit may determine that the voice display condition is satisfied when it detects that a predetermined word has been spoken.
[0019] Examples of "predetermined words" include words related to the recipient's personal information (such as name, department name, job responsibilities, etc.), words related to date and time, and words prompting confirmation such as "Is this okay?".
[0020] In this way, when a specific word is spoken, the user's voice is converted into text and displayed on the display, thereby realizing a videoconferencing system that displays text at a more appropriate time than before so that the listener can easily focus on it.
[0021] The control unit may determine that the condition for audio display is satisfied when it detects that a predetermined button is pressed.
[0022] Examples of the "predetermined button" include a "start missed broadcast function" button and a "start attention function" button. Note that the audio data may be displayed when a predetermined condition is satisfied after a predetermined button is pressed.
[0023] In this way, when a specific button is pressed, the user's voice is converted into text and displayed on the display, making it possible to realize a videoconferencing system that displays text at a more appropriate time than before so that the listener can easily focus on it.
[0024] The present disclosure will be described in more detail below with reference to the accompanying drawings. Note that the following description is illustrative in all respects and should not be construed as limiting the present disclosure.
[0025] [Embodiment 1] 1 to 3, a schematic configuration of a video conference system 100 including user terminals 1A to 1D which is an embodiment of an image display device according to the present disclosure will be described.
[0026] FIG. 1 is an explanatory diagram showing an example of the configuration of a video conference system 100 according to the present disclosure.
[0027] As shown in FIG. 1, a video conference system 100 according to this disclosure is made up of user terminals 1A to 1D and a server 2 connected via a network 3. The user terminals 1A to 1D exchange information in real time via the server 2.
[0028] In the following description, the user terminals 1A to 1D will be collectively referred to as the user terminal 1.
[0029] <Overview of user terminal 1> Next, the schematic configuration of the user terminal 1 will be described with reference to FIG.
[0030] FIG. 2 is a block diagram showing a schematic configuration of the user terminal 1 of FIG. As shown in FIG. 2, the user terminal 1 includes a control unit 10, a memory unit 11, an image processing unit 12, a communication unit 13, a display unit 14, an operation unit 15, an imaging unit 16, an audio acquisition unit 17, an audio output unit 18 and an audio display unit 19.
[0031] Each component of the user terminal 1 will now be described.
[0032] The control unit 10 comprehensively controls the user terminal 1, and is composed of a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), various interface circuits, and the like.
[0033] The control unit 10 monitors and controls all loads such as the detection of each sensor and the operation unit 15 in order to control the operation of the entire user terminal 1 .
[0034] The storage unit 11 is an element or storage medium that stores information and control programs required to realize various functions of the user terminal 1. For example, a semiconductor element such as a RAM or a ROM, a hard disk, a flash storage unit, an SSD (Solid State Drive), or other storage medium is used.
[0035] The program and data may be stored in different devices, such as by configuring the area for storing data in a hard disk drive and the area for storing programs in a flash memory unit.
[0036] The image processing unit 12 is a part that converts image data into an appropriate electrical signal and performs processing such as enlargement / reduction so as to make it suitable for output.
[0037] The communication unit 13 communicates with other user terminals 1 and external servers 2 via the network 3, and transmits and receives various data such as image data of materials and documents to be displayed during the video conference, image data of the user's face, and audio data.
[0038] The display unit 14 is a part that displays various types of information of the user terminal 1. The display unit 14 is, for example, a CRT display, a liquid crystal display, an EL display, or the like, and is a display device such as a monitor or line display for displaying electronic data such as the processing status of the operating system and application software.
[0039] The control unit 10 displays the operation and status of the user terminal 1 through the display unit 14.
[0040] The operation unit 15 is an interface for operating the user terminal 1, and is a part that receives various commands from the user.
[0041] In addition, the display unit 14 and the operation unit 15 may be composed of a display panel composed of a liquid crystal panel or the like, and a touch panel of a capacitive type or the like that is placed on top of the display panel and detects the position touched by a finger.
[0042] The imaging unit 16 is a part that captures an image of the face and movements of the user of the user terminal 1 to obtain an image of the user's face and movements, and may be, for example, a camera.
[0043] The voice acquisition unit 17 is a part that acquires the user's voice or sound (hereinafter collectively referred to as voice) inputted by the microphone of the user terminal 1. Furthermore, the voice acquisition unit 17 acquires the voices of other users from the other user terminals 1 via the communication unit 13 .
[0044] The audio output unit 18 is a part that outputs audio from video applications, audio applications, and the like, and is implemented by using a speaker, for example.
[0045] The audio display unit 19 is a part that converts the user's audio data input through the microphone of the user terminal 1 and other users' audio data obtained from other user terminals 1 via the communication unit 13 into text data and displays it on the display unit 14.
[0046] <Server 2 outline configuration> Next, the schematic configuration of the server 2 will be described with reference to FIG.
[0047] FIG. 3 is a block diagram showing a schematic configuration of the server 2 in FIG. As shown in FIG. 3, the server 2 includes a control unit 20, a storage unit 21, an image processing unit 22, a communication unit 23, a display unit 24, an operation unit 25, and a video conference system management unit .
[0048] Each component of the server 2 will be described below.
[0049] The control unit 20 controls the server 2 in an integrated manner, and is composed of a CPU, RAM, ROM, various interface circuits, and the like.
[0050] The control unit 20 monitors and controls all loads such as the detection of each sensor and the operation unit 25 in order to control the operation of the entire server 2 .
[0051] The storage unit 21 is an element or storage medium that stores information and control programs required to realize various functions of the server 2. For example, a semiconductor element such as a RAM or a ROM, a hard disk, a flash storage unit, an SSD, or other storage medium is used.
[0052] The storage unit 21 stores data necessary for carrying out a video conference, such as information related to the video conference system 100 and image data. Moreover, the storage unit 21 stores predetermined audio display conditions.
[0053] The program and data may be stored in different devices, such as by configuring the area for storing data in a hard disk drive and the area for storing programs in a flash memory unit.
[0054] The image processing unit 22 is a part that converts image data into an appropriate electrical signal and performs processing such as enlargement / reduction so as to make it suitable for output.
[0055] The communication unit 23 communicates with the external user terminal 1 via the network 3, and transmits and receives material data of the video conference system 100 and the like.
[0056] The display unit 24 is a part that displays various types of information about the server 2. The display unit 24 is, for example, a CRT display, a liquid crystal display, an EL display, or the like, and is a display device such as a monitor or line display for displaying electronic data such as the processing status of the operating system and application software.
[0057] The control unit 20 displays the operation and status of the server 2 through the display unit 24 .
[0058] The operation unit 25 is an interface for operating the server 2, and is a part that receives commands from the user.
[0059] In addition, the display unit 24 and the operation unit 25 may be composed of a display panel composed of a liquid crystal panel or the like, and a touch panel of a capacitive type or the like that is placed on top of the display panel and detects the position touched by a finger.
[0060] The video conference system management unit 26 is a part that manages a system for carrying out a video conference among a plurality of user terminals 1 via the network 3 .
[0061] <An example of the audio data display process of the video conference system 100 according to the first embodiment of the present disclosure> Next, an example of a voice data display process of the video conference system 100 according to the first embodiment of the present disclosure will be described with reference to FIG. 4 and FIG.
[0062] FIG. 4 is a flowchart showing an example of a voice data display process of the video conference system 100 of FIG.
[0063] After the video conference has started, in step S1 of FIG. 4, the control unit 10 of the user terminal 1 starts camera image recognition to cause the imaging unit 16 to capture the user's face and movements, and starts voice input detection to cause the voice capture unit 17 to capture the user's voice (step S1).
[0064] Next, in step S2, the control unit 10 determines whether or not the missed broadcast function start button displayed on the display unit 14 has been pressed (step S2).
[0065] FIG. 5 shows an example of a video conference screen displayed on the display unit 14 of the user terminal 1 in the video conference system 100 of FIG. FIG. 5(A) shows an example of the video conference screen before the missed call function start button is pressed.
[0066] In the example of FIG. 5, user A, who is the speaker, is displayed on the large screen (main screen), and users B, C, D, and E, who are the listeners, are displayed on small screens (sub-screens) on the right side of the screen.
[0067] As shown in FIG. 5(A), a "Start Missed Broadcast Function" button and a "Settings" button are displayed in the upper left portion of the video conference screen displayed on the display unit 14 of the user terminal 1.
[0068] If the missed broadcast function start button has not been pressed (if the determination in step S2 is No), in step S7, the control unit 10 determines whether or not the video conference has ended (step S7).
[0069] When the video conference is ended (when the determination in step S7 is Yes), the control unit 10 ends the process. On the other hand, if the video conference has not ended (if the determination in step S7 is No), the control unit 10 returns the process to the determination in step S2.
[0070] When the missed broadcast function start button is pressed (when the determination in step S2 is Yes), in step S3, the control unit 10 determines whether or not a predetermined audio data display condition is satisfied (step S3).
[0071] An example of a condition for displaying audio data is when the recipient user removes the headphones. The details of the audio data display conditions will be described later in the description of FIGS.
[0072] If the voice data display condition is satisfied (if the determination in step S3 is Yes), in step S4, the control unit 10 starts displaying the voice data (step S4).
[0073] FIG. 5(B) shows an example of the video conference screen after the missed call function start button is pressed. As shown in Figure 5(B), when the receiver, user C, removes the headphones at 14:05, the voice data of the speaker, user A, "2:05 PM: [Mr. A] Let me explain about XX, one of today's agenda items," is displayed on the screen of user C's user terminal 1 as it is possible that he or she may have missed the conversation. Additionally, the detection time of the audio data, "14:05," is displayed on the slide bar.
[0074] The audio data display is only displayed on the user terminal 1 of user C who has removed the headphones.
[0075] If the conditions for displaying audio data remain met, such as if the user continues to remove their headphones, audio data will continue to be displayed in real time in accordance with the ongoing conference.
[0076] On the other hand, if the audio data display condition is not satisfied (if the determination in step S3 is No), the control unit 10 performs the determination in step S5.
[0077] Next, in step S5, the control unit 10 determines whether or not the missed broadcast function end button has been pressed (step S5).
[0078] As shown in FIG. 5(B), after the miss-listen function start button is pressed, the miss-listen function start button changes into a miss-listen function end button. The missed broadcast function end button may be highlighted so that the user can easily recognize that the missed broadcast function is in progress.
[0079] If the missed broadcast function end button is pressed (if the determination in step S5 is Yes), in step S6, the control unit 10 ends the display of the audio data (step S6). Thereafter, the control unit 10 returns the process to the determination in step S2.
[0080] <Example of setting the missed call function of the video conference system 100 according to the first embodiment of the present disclosure> Next, a setting example of the missed call function of the video conference system 100 according to the first embodiment of the present disclosure will be described with reference to FIGS. FIG. 6 shows an example of a setting screen displayed when a setting button for the missed call function displayed on the display unit 14 of the user terminal 1 in the video conference system 100 of FIG. 1 is pressed.
[0081] When the user presses the "Settings" button as shown in FIG. 6(A), a setting screen for the missed broadcast function is displayed as a sub-screen as shown in FIG. 6(B).
[0082] FIG. 7 shows an example of a setting screen for the missed-message function displayed on the display unit 14 of the user terminal 1 in the video conference system 100 of FIG.
[0083] In the example of FIG. 7, setting buttons for setting the audio data display conditions for the missed audio function, that is, "action", "person", "word" and "environment" buttons are displayed. Also displayed are a "History" button for automatically setting up based on the history of past video conferences, etc., and a "Done" button for ending the setting up.
[0084] FIG. 8 shows an example of an operation setting screen when the "Operation" button is selected on the setting screen for the missed program function in FIG. The operation setting screen is displayed as a sub-screen, similar to the setting screen in FIG. 6(B). The same applies to the other setting screens.
[0085] When the "Action" button is selected on the setting screen of FIG. 7, the screen of FIG. 8 will be displayed. In the example of FIG. 8, the following actions are set: Action 1 "remove headphones / earphones", Action 2 "leave seat", and Action 3 "look away (for one minute or more)".
[0086] When it is detected that the listener has performed any one of the actions 1 to 3 after the missed call function start button is pressed, the audio data display starts. Also, the display of the voice data may be terminated when the listener puts the head-on earphones back on, sits down, or looks back.
[0087] The start and end of the audio data do not necessarily have to coincide with the time when the missed-listen function start button and end button are pressed. For example, in order to prevent the speaker's speech from being interrupted while the listener has removed the headphones, the speech data may be configured to display the speaker's entire speech even if the time is slightly off.
[0088] The user can also press the triangle button to the right of the action list to display a drop-down list of pre-defined actions, select a desired action from these, and then press the "Done" button at the bottom right to set the action. Alternatively, the user may directly enter the desired action into a form.
[0089] FIG. 9 shows an example of a person setting screen when the "People" button is selected on the setting screen for the missed program function in FIG.
[0090] When the "People" button is selected on the settings screen of FIG. 7, the screen of FIG. 9 will be displayed. In the example of FIG. 9, name 1 "Director Yamada", name 2 "Section Manager Kato", and name 3 "Person in charge Tanaka" are set.
[0091] The list of names is automatically extracted from the conference information (conference name, participants, department names, conference materials, etc.), and the user can select the desired name from a drop-down list.
[0092] After the missed call function start button is pressed, when any of the speakers named 1 to 3 speaks, the voice data display starts.
[0093] It should be noted that the start and end of the voice data do not necessarily need to coincide with the time when the speaker of the target name is speaking. For example, when other people ask questions before or during a speaker's speech, the speaker's speech including those questions may be displayed together in the voice data.
[0094] In this way, a videoconferencing system 100 can be realized that allows the recipient, such as a superior, colleague, or related party, to set the name of a person whom they do not want to miss, and then displays the audio data when that person speaks, thereby improving the ability to not miss a call.
[0095] FIG. 10 shows an example of a word setting screen when the "word" button is selected on the setting screen for the missed program function in FIG.
[0096] When you select the "Words" button on the settings screen in Figure 7, the screen in Figure 10 will be displayed. In the example of FIG. 10, word 1 "your name," word 2 "date and time," and word 3 "is this OK?" are set.
[0097] Here, if "self" is set, words in the recipient's own information (such as name, department name, and job duties) are used as the judgment conditions. If "date and time" is set, words related to the date and time, such as "MM month DD day" or "next week", are used as the judgment criteria.
[0098] After the missed speech function start button is pressed, when the speaker makes an utterance that includes any one of these words 1 to 3, the voice data display starts.
[0099] In addition, if the word "Is that okay?" is set, in addition to that word, synonyms with similar meanings such as "Is that okay?", "Does it matter?", and "Is that possible?" will also be included in the judgment criteria.
[0100] The word list is a list of words automatically extracted from the meeting information (meeting name, participants, department names, meeting materials, etc.), and allows the user to select the desired word from a drop-down list. Alternatively, the user may enter the desired words directly into the form.
[0101] FIG. 11 shows an example of an environment setting screen when the "Environment" button is selected on the setting screen for the missed program function in FIG.
[0102] When you select the "Environment" button on the settings screen in Figure 7, the screen in Figure 11 will be displayed. In the example of FIG. 11, environment 1 "speaker muted" and environment 2 "degraded communication quality" are set.
[0103] Here, the case of "deterioration in communication quality" refers to a case where the communication environment on the receiver's side is so bad that screen display and audio output are not possible or are unstable. However, this is limited to cases where there are no problems with the communication environment on the speaker's side and all of the audio data of the speaker's utterance can be displayed.
[0104] In the case of "decline in communication quality," since it may not be possible to display the voice data in real time, it may be possible to display the voice data from the period of degraded communication quality as a history after the communication quality is restored.
[0105] The environment list is a list of assumed environmental conditions, and allows the user to select a desired environmental condition from a drop-down list. Alternatively, the user may directly input the desired environmental conditions into a form.
[0106] FIG. 12 shows an example of a history setting screen when the "History" button is selected on the setting screen for the missed program function in FIG.
[0107] When you select the "History" button on the settings screen in Figure 7, the screen in Figure 12 will be displayed. In the example of FIG. 12, history 1 "20Y1, M1, D1, 'Sales Meeting'" and history 2 "20Y2, M2, D2, 'Regular Reporting Session'" are set.
[0108] In this case, the judgment conditions (actions, people, words, and environment) set in these past meetings and reports will be reflected in the judgment conditions.
[0109] In this way, when a past conference or report session related to the current video conference contains similar items that the listeners want to avoid missing, it is possible to save the trouble of having to set the conditions for each item each time.
[0110] Settings can be made by combining these "Action," "People," "Words," "Environment," and "History" buttons. By setting multiple conditions in this way in any combination, it is possible to pinpoint information that the listener does not want to miss.
[0111] These settings may be applied only to the user terminal 1 of an individual listener, or may be applied commonly to multiple user terminals 1 by an administrative user, or a combination of both.
[0112] <Example of display of missed broadcast results history in the video conference system 100 according to the first embodiment of the present disclosure> Next, an example of a history display of missed messages in the video conference system 100 according to the first embodiment of the present disclosure will be described with reference to FIGS.
[0113] FIG. 13 shows an example of a history confirmation display of missed conversation results on the video conference screen displayed on the display unit 14 of the user terminal 1 in the video conference system 100 of FIG.
[0114] If there is a possibility that a missed program has occurred, the control unit 10 displays a history of missed programs for confirmation. In the example of FIG. 13, the message "Do you want to check the speech history for the period when the headphones were off?" is displayed along with "Yes" and "No" buttons.
[0115] FIG. 14 shows an example of a history display of missed conversation results on the video conference screen displayed on the display unit 14 of the user terminal 1 in the video conference system 100 of FIG.
[0116] In the example of FIG. 14, it is assumed that the words "user" and "date and time" are set as the criteria for determining whether a message has been missed in the user terminal 1 of user C, who is the listener.
[0117] In this case, as shown in FIG. 14, the voice data history such as "XX will be handled by 'Mr. C' and 'Mr. E'" and "Each person should prepare the YY document by the end of next week" is displayed.
[0118] Also, when the user presses the "End Display" button at the bottom right of the screen, the screen returns to its original configuration.
[0119] FIG. 15 shows another example of a history display of missed conversation results on the video conference screen displayed on the display unit 14 of the user terminal 1 in the video conference system 100 of FIG.
[0120] In the example of FIG. 15, it is assumed that the word "myself" is set as a criterion for determining whether or not a message has been missed in the user terminal 1 of the listener, user C.
[0121] In this case, as shown in FIG. 15, the history of the audio data is displayed in a timeline, and the times when the missed audio occurred ("2:13" and "2:25") are highlighted with red circles. When User C places the mouse cursor over the highlighted red circle, the audio data history is displayed, showing "XX will be handled by Mr. C and Mr. E," making it easy to check the content of the parts that were missed.
[0122] In this way, when a specific voice display condition is satisfied, the voice of the speaking user is converted into text and displayed, thereby realizing a videoconferencing system 100 that displays text at a more appropriate timing than before so that the listener can easily focus on it.
[0123] [Embodiment 2] <An example of the audio data display process of the video conference system 100 according to the second embodiment of the present disclosure> Next, an example of a voice data display process of the video conference system 100 according to the second embodiment of the present disclosure will be described with reference to FIGS.
[0124] The first embodiment is characterized in that, in a video conference, when a listener presses a missed message function start button and a predetermined condition is satisfied, audio data display starts.
[0125] In contrast, in the second embodiment, a case will be described in which the speaker starts displaying voice data on the user terminal 1 of the listener.
[0126] The schematic configuration of the video conference system 100 according to the second embodiment of the present disclosure is the same as that of the first embodiment (FIGS. 1 to 3), and therefore a description thereof will be omitted.
[0127] FIG. 16 is a flowchart showing an example of an audio data display process of the video conference system 100 according to the second embodiment of the present disclosure.
[0128] After the user terminal 1 starts a video conference, in step S11 of FIG. 16, the control unit 10 of the user terminal 1 starts camera image recognition to cause the imaging unit 16 to capture an image of the user's face, and starts voice input detection to cause the voice acquisition unit 17 to acquire the user's voice (step S11).
[0129] Next, in step S12, the control unit 10 determines whether or not the noted function start button has been pressed (step S12).
[0130] FIG. 17 shows an example of a video conference screen displayed on the display unit 14 of the user terminal 1 in the video conference system according to the second embodiment of the present disclosure.
[0131] FIG. 17(A) shows an example of the video conference screen before the featured function start button is pressed.
[0132] In the example of FIG. 17, user A, who is the speaker, is displayed on the large screen, and users B, C, D, and E, who are the listeners, are displayed on small screens on the right side of the screen.
[0133] As shown in FIG. 17(A), "Start featured function" is displayed in the upper left portion of the video conference screen displayed on the display unit 14 of the user terminal 1.
[0134] If the noted function start button has not been pressed (if the determination in step S12 is No), in step S16, the control unit 10 determines whether or not the video conference has ended (step S16).
[0135] When the video conference is ended (when the determination in step S16 is Yes), the control unit 10 ends the process. On the other hand, if the video conference has not ended (if the determination in step S16 is No), the control unit 10 returns the process to the determination in step S12.
[0136] On the other hand, if the noted function start button has been pressed (if the determination in step S12 is Yes), in step S13, the control unit 10 starts displaying the audio data (step S13).
[0137] FIG. 17(B) shows an example of the video conference screen after the featured function start button is pressed. As shown in Figure 1(B), the voice data of the speaker, User A, "PM2:05: [Mr. A] I will explain about XX, one of today's agenda items," is displayed on the screen.
[0138] Next, in step S14, the control unit 10 determines whether or not the noted function end button has been pressed (step S14). If the noted function end button is pressed (if the determination in step S14 is Yes), in step S15, the control unit 10 ends the audio data display (step S15). Thereafter, the control unit 10 returns the process to the determination in step S12.
[0139] Note that the attention function start button does not necessarily have to be pressed by the speaker, but may also be pressed by the listener. Also, one of the users may press a focus function start button to draw the attention of other users to important utterances.
[0140] In this way, when a speaker presses the focus function start button, the speaker's voice is converted into text and displayed, thereby realizing a videoconferencing system 100 that displays text at a more appropriate timing than before so that the listener can easily focus on it.
[0141] [Embodiment 3] <An example of the audio data display process of the video conference system 100 according to the third embodiment of the present disclosure> Next, an example of a voice data display process of the video conference system 100 according to the third embodiment of the present disclosure will be described with reference to FIGS.
[0142] FIG. 18 shows an example of a setting screen for the missed call function displayed on the display unit 14 of the user terminal 1 in the video conference system 100 according to the third embodiment of the present disclosure.
[0143] In the example of FIG. 18, setting buttons for setting the audio data display conditions for the missed-listen function, that is, the "action", "person", "word", and "environment" buttons as well as the "speech" button are displayed.
[0144] FIG. 19 shows an example of an operation setting screen when the "Speak" button is selected on the setting screen for the missed call function in FIG.
[0145] When the "Speech" button is selected on the setting screen of FIG. 18, the screen of FIG. 19 is displayed. In the example of Figure 19, utterance 1 "The speaker's voice is low", utterance 2 "Multiple speakers spoke at the same time", utterance 3 "The speaker is wearing a mask", and utterance 4 "The speaker is displaying materials" are set.
[0146] FIG. 20 shows an example of the display of audio data on a video conference screen displayed on the display unit 14 of the user terminal 1 in the video conference system 100 according to the third embodiment of the present disclosure.
[0147] As shown in the example of FIG. 20, when the voice of user A who is a speaker is lower than a predetermined volume, voice data saying "Today's topic is XX" is displayed.
[0148] FIG. 21 shows an example of the audio data display on the video conference screen displayed on the display unit 14 of the user terminal 1 when a plurality of speakers speak simultaneously in the video conference system 100 according to the third embodiment of the present disclosure.
[0149] In FIG. 21, it is assumed that three users A, B, and D speak simultaneously. In this case, as shown in the example of FIG. 21(A), only users A, B, and D who spoke simultaneously are displayed on a divided screen, and the voice data of users A, B, and D are displayed respectively.
[0150] Then, the display of the voice data ends when the user presses the "end display" button or when the next speaker speaks.
[0151] Also, as shown in the example of FIG. 21(B), the voice data of users A, B, and D who simultaneously spoke may be displayed in the order in which they spoke, while maintaining the current screen configuration of the video conference. When a plurality of users speaks at approximately the same time, the voice data is displayed in a predetermined order of users, such as alphabetical order.
[0152] FIG. 22 shows an example of the audio data displayed on the video conference screen displayed on the display unit 14 of the user terminal 1 when the speaker is wearing a mask in the video conference system 100 according to the third embodiment of the present disclosure.
[0153] As shown in the example of FIG. 22, when a speaker wearing a mask is detected, the voice data is displayed. In this case, the voice text may be displayed in a speech bubble from the speaker's mouth.
[0154] In addition, the speech bubble may be displayed to clearly indicate who is speaking not only when a mask is being worn, but also when other conditions are met, such as a low voice or simultaneous speech.
[0155] FIG. 23 shows an example of audio data displayed on the video conference screen displayed on the display unit 14 of the user terminal 1 when a speaker is speaking while displaying a document in the video conference system 100 according to the third embodiment of the present disclosure.
[0156] When the speaker, User A, is speaking while displaying a document, the audio data is displayed enlarged or reduced and written vertically or horizontally so as not to overlap with the text or figures in the document, as shown in the example of Figure 23(A).
[0157] Then, the display of the voice data ends when the user presses the "end display" button or when the next speaker speaks.
[0158] Furthermore, as shown in the example of FIG. 23(B), when the audio data cannot be displayed on the main screen due to the document being enlarged, the audio data may be displayed on a sub-screen next to it.
[0159] In this way, when a speaker satisfies a specified voice display condition, the voice of the user who satisfies the voice display condition is converted into text and displayed, thereby realizing a videoconferencing system 100 that displays text at a more appropriate timing than before so that it is easier for the listener to focus on.
[0160] Preferred aspects of the present disclosure include any combination of the above-mentioned aspects.
[0161] In addition to the above-mentioned embodiment, various modifications of this disclosure are possible. These modifications should not be interpreted as not belonging to the scope of this disclosure. This disclosure should include all modifications within the scope of the claims and the equivalent meanings. [Explanation of symbols]
[0162] 1, 1A, 1B, 1C, 1D: user terminal, 2: server, 3: network, 10, 20: control section, 11, 21: storage section, 12, 22: image processing section, 13, 23: communication section, 14, 24: display section, 15, 25: operation section, 16: imaging section, 17: audio acquisition section, 18: audio output section, 19: audio display section, 26: video conference system management section, 100: video conference system
Claims
1. A video conference system in which images and audio are shared via a plurality of user terminals connected through a network, the user terminal comprises an imaging unit which captures an image of the user's face, an audio acquisition unit which acquires the voice of the user, a communication unit which transmits and receives various data including image and audio data to and from other user terminals, a display unit which displays the image of the user's face on a video conference screen, an audio output unit which outputs the voice of the user, an audio display unit which converts the voice of the user into text and displays it on the display unit, and a control unit which controls the imaging unit, the audio acquisition unit, the communication unit, the display unit, the audio output unit, and the audio display unit; A video conferencing system characterized in that, when predetermined voice display conditions are met during a video conference, the control unit causes the voice display unit to convert the user's voice being spoken into text and display it on the display unit.
2. The imaging unit captures an image of the user's movement, The video conference system according to claim 1 , wherein the control unit determines that the voice display condition is satisfied when it detects that the user, who is a listener, has performed a predetermined action.
3. The video conference system according to claim 1 , wherein the control unit determines that the voice display condition is satisfied when it detects that a predetermined user has spoken.
4. The video conference system according to claim 1 , wherein the control unit determines that the voice display condition is satisfied when it detects that a predetermined word has been spoken.
5. The video conference system according to claim 1 , wherein the control unit determines that the audio display condition is satisfied when it detects that a predetermined button is pressed.
Citation Information
Patent Citations
Voice recognition system, voice recognition method and program
JP2022142374A