Information terminal, voice processing method for information terminal, and conference system

Abstract

Provided are an information terminal, a voice processing method for the information terminal and a WEB conference system that make it possible to hold a WEB conference using stereophonic sound signals, and also reproduce a WEB conference using stereophonic sound signals. The present invention is configured such that: a head-related transfer function of a participant, who is included in a participant list, is determined with reference to a head-related transfer function database by using position information of the participant in a virtual space; speech audio at a WEB conference is converted into a stereophonic sound signal on the basis of the determined head-related transfer function, and the stereophonic sound signal is outputted; a transcription file having a dataset recorded therein for each spoken word of the participant is generated, on the basis of the speech audio at the WEB conference, and by using a dataset which has, as constituents thereof, text data of the speech audio divided by markup tags, speaker information, and a time stamp of a conference elapsed time; and the transcription file is stored.

Description

Information terminal, audio processing method for information terminal, and conference system

[0001] The present invention relates to an information terminal such as a personal computer, a smartphone, or a wearable display that can use a web conferencing application, an audio processing method for the information terminal, and a conferencing system.

[0002] Patent Document 1 discloses a voice output device that simply and conveniently changes and outputs a voice that reads out text data. Specifically, the voice output device modulates and reads out a voice signal in accordance with attribute information, such as character decoration, of character strings that make up the text data.

[0003] JP 2016-170218 A

[0004] In recent years, web conferences have been held between remote locations using information terminals such as personal computers, smartphones, and wearable displays such as head-mounted displays that can use web conference applications. In this specification, personal computers are also referred to as PCs, and head-mounted displays are also referred to as HMDs. Web conference applications may have a transcription function. This makes it possible to convert speech during web conferences into text and display it to improve accessibility, or to save the text data of speech and use it as meeting minutes.

[0005] In addition, web conferencing applications may have a function to read saved text data aloud using speech synthesis processing, a technology that mechanically creates a voice that reads the text aloud. This makes it easier to check minutes, etc. Furthermore, information terminals themselves may have a function to read text data aloud as a basic function. In other words, the mutual conversion between text and speech is used not only in web conferencing applications but also in various fields.

[0006] Meanwhile, functions for reproducing stereophonic signals are becoming increasingly common in information terminals. The use of stereophonic signals is expected to overcome, for example, the weakness of web conferences, such as a lack of realism. While the technology disclosed in Patent Document 1 can add intonation to audio signals, this does not necessarily result in a more realistic experience, in other words, a conference that is closer to a face-to-face conference.

[0007] The present invention has been made in view of the above, and one of its objects is to provide an information terminal, an audio processing method for an information terminal, and a web conference system that can hold a web conference using a stereophonic signal and further reproduce a web conference using a stereophonic signal.

[0008] The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

[0009] In one embodiment, the system stores a participant list representing information about participants in a web conference and a database of head-related transfer functions for generating stereophonic signals, displays an image of the virtual space in which the web conference is held, determines the head-related transfer functions of the participants by referring to the database using the position information of the participants in the virtual space contained in the participant list, converts the speech in the web conference into a stereophonic signal based on the determined head-related transfer functions and outputs it, generates a transcription file that records a data set for each participant's speech based on the speech in the web conference using a data set whose components are text data of the speech separated by markup tags, speaker information, and a timestamp of the elapsed time of the conference, and stores the transcription file.

[0010] According to one embodiment, a web conference using a stereophonic signal can be held, and further, a web conference using a stereophonic signal can be reproduced.

[0011] 1 is a block diagram showing an example of the configuration of an information terminal capable of executing a web conference application in one embodiment. FIG. 2 is a schematic diagram showing an example of the configuration of a web conference system according to one embodiment. FIG. 3 is a sequence diagram showing an example of processing content performed between the web conference application of the information terminal shown in FIG. 1 and the web conference service of the web conference server shown in FIG. 2. FIG. 4 is a diagram showing an example of the contents of the participant list in FIGS. 1 and 3. FIG. 5 is a diagram showing an example of an updated image of the virtual conference room associated with step S14 in FIG. 3. FIG. 6 is a flowchart showing an example of processing content based on the web conference application of the information terminal shown in FIG. 1 in a first embodiment. FIG. 7 is a diagram showing an example of the configuration of a transcription file generated by the flowchart shown in FIG. 6. FIG. 8 is a flowchart showing an example of processing content when reproducing the transcription file shown in FIG. 7. FIG. 9 is a flowchart showing an example of detailed processing content associated with outputting a stereophonic signal in FIG. 6 or FIG. 8. FIG. 10 is a diagram showing an example of an image of the virtual conference room obtained by modifying FIG. 5. FIG. 11 is a flowchart showing an example of processing content based on the web conference application of the information terminal shown in FIG. 1 in a second embodiment. FIG. 12 is a diagram showing an example of the configuration of a transcription file generated by the flowchart shown in FIG. 13 is a flowchart showing an example of processing content based on a web conference application of the information terminal shown in FIG. 1 in a third embodiment. FIG. 14 is a diagram showing an example of the configuration of a transcription file generated by the flowchart shown in FIG. 13. FIG. 15 is a flowchart showing an example of processing content based on a web conference application of the information terminal shown in FIG. 1 in a fourth embodiment. FIG. 16 is a diagram showing an example of the configuration of a transcription file generated by the flowchart shown in FIG. 15. FIG. 17 is a flowchart showing an example of processing content based on a web conference application of the information terminal shown in FIG. 1 in a fifth embodiment. FIG. 18 is a diagram showing an example of the configuration of a transcription file generated by the flowchart shown in FIG. 17. FIG. 19 is a flowchart showing an example of processing content based on a web conference application of the information terminal shown in FIG. 1 in a sixth embodiment.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all drawings used to explain the embodiment, the same components are generally designated by the same reference numerals, and repeated explanations will be omitted. In the following description, the term "web conference" may be replaced with "online conference" or "video conference."

[0013] Furthermore, by using the audio playback technology described in the embodiment, it is possible to contribute to "9. Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation" of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0014] <Outline of Information Terminal> Figure 1 is a block diagram showing an example of the configuration of an information terminal capable of executing a web conferencing application in one embodiment. The information terminal 1 shown in Figure 1 is, for example, a PC, a mobile terminal such as a smartphone, or a wearable display such as an HMD. The information terminal 1 includes a camera / sensor unit 10, an image display unit 11, an operation input unit 12, an audio input unit 13, an audio output unit 14, a communication unit 15, a main processing unit 16, a memory unit 17, and a storage unit 18.

[0015] The main processing unit 16 is configured with a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), etc. The memory unit 17 is one of storage devices and is configured with, for example, a RAM or a flash memory.

[0016] The memory unit 17 stores an operating system (OS) and programs such as operation control applications used by the main calculation unit 16. The processor constituting the main calculation unit 16 controls each component and controls the operation of the information terminal 1 as a whole by executing the OS, operation control applications, and other programs stored in a storage device, specifically, in the memory unit 17.

[0017] The memory unit 17 also stores various types of information data, such as data of output audio signals generated by the information terminal 1, data of three-dimensional images in virtual reality (VR) and augmented reality (AR), and data of stereophonic content. In other words, the memory unit 17 also serves as a working RAM for the main processing unit 16.

[0018] The storage unit 18 is another type of storage device and is configured, for example, by a non-volatile storage medium such as a flash ROM (also referred to as FROM). The storage unit 18 stores a basic operation program 20, a web conference application 21, and data 22. The basic operation program 20 is an OS or the like, and the web conference application 21 is a program for participating in a web conference. The basic operation program 20 and the web conference application 21 are loaded into the memory unit 17 and executed by the main processing unit 16.

[0019] The data 22 includes various data required when executing the basic operation program 20 and when executing the web conference application 21. The various data include a participant list 23, a database (referred to as D / B) 24 of head-related transfer functions for generating stereophonic signals, and a transcription file 25. The participant list 23 represents information about participants in the web conference. The transcription file 25 is generated using the web conference application 21.

[0020] The camera and sensor unit 10 includes a camera that captures images of the user of the information terminal 1 and a group of sensors, such as an illuminance sensor, a proximity sensor, a distance sensor, and a motion sensor, that assist in the use of the information terminal 1. The camera may capture images of the scenery in front of the user as well as the user. The camera converts light input through the lens into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The camera then generates visible image data of the surroundings and objects based on the electrical signal.

[0021] Alternatively, a time-of-flight (TOF) sensor may be provided instead of or in combination with the camera, capable of acquiring the distance to an object being imaged as a distance image. Using these visible and distance images, for example, pointing actions, such as those used when making a selection in multiple virtual reality spaces displayed on an information terminal 1 such as an HMD, can be accurately detected. A pointing action is when a user holds their hand over an image of a virtual reality space and pinches their thumb and index finger together. Alternatively, a camera can be used to scan the surroundings, and a three-dimensional space map can be generated using the outputs of the above-mentioned sensors and the camera.

[0022] Furthermore, the sensor unit may include an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc. The acceleration sensor detects acceleration, which is the change in speed per second, and detects movement, vibration, shock, etc. The gyro sensor detects angular velocity in the direction of rotation and detects vertical, horizontal, and diagonal orientations. The geomagnetic sensor detects the Earth's magnetic force and detects the direction in which the information terminal 1 is facing. In this way, by using motion sensors such as an acceleration sensor, gyro sensor, or geomagnetic sensor, the position, tilt, direction, movement, etc. of the information terminal 1 can be detected.

[0023] Specifically, for example, if the information terminal 1 is a wearable display, a gyro sensor or a geomagnetic sensor can be used, and an acceleration sensor can also be used if necessary, to detect the movement of the user's head. In particular, if a three-axis geomagnetic sensor that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions is used, head movement can be detected with higher accuracy by detecting changes in geomagnetic field in response to head movement. Furthermore, if a device paired with these sensors is worn on the user's hand or arm, hand or arm movement can also be detected.

[0024] The image display unit 11 displays an image of a virtual conference room received from a web conference server (described later) and avatar images of web conference participants. The virtual conference room is a virtual space where the web conference is held. The image display unit 11 may also display virtual reality (VR) information, real space information captured by the camera 10, and the like. The image display unit 11 is configured with various displays, such as an organic EL display or a liquid crystal display, and displays real space information, virtual reality (VR) information, and augmented reality (AR) information as images.

[0025] Furthermore, for example, in an information terminal 1 such as an HMD, the image display unit 11 may display notification information to the user on a menu screen or the like, or the operating status of the HMD, etc. Furthermore, the image display unit 11 may display information to notify the user, for example, when starting, pausing, or resuming the emission of an output audio signal. This allows the user to recognize that, for example, when the emission of sound is paused or resumed, the pause or resumption is due to a normal control operation and not a malfunction.

[0026] The operation input unit 12 is a user interface that accepts operation inputs from a user. In an information terminal 1 such as a PC, the operation input unit 12 is composed of a keyboard, a mouse, etc., and in an information terminal 1 such as a smartphone, the operation input unit 12 is composed of a touch panel with a touch sensor incorporated into the flat display that is the image display unit 11. In an information terminal 1 such as a wearable display, the operation input unit 12 is realized by a virtual keyboard and virtual buttons displayed as virtual objects.

[0027] The audio input unit 13 is a microphone or the like, and collects the user's speech. For example, the audio input unit 13 collects the user's speech when the user is in a web conference, on the telephone or online call, or talking with others in real space. The audio output unit 14 is connected to a sound emitting device and includes an audio output interface that interfaces with the sound emitting device. The audio output unit 14 outputs the speech or an audio signal thereof, for example, during a web conference or the like.

[0028] The sound emitting device is, for example, a speaker or headphones. The sound emitting device is provided inside or outside the audio output unit 14. For example, a sound emitting device such as a speaker is provided inside the audio output unit 14. On the other hand, if the sound emitting device is headphones or the like, the audio output unit 14 is provided with an audio output terminal that can be connected to external headphones via an audio output interface.

[0029] Here, when the speech sound to be output is sound using a stereophonic signal, the sound emitting device is preferably headphones. The headphones convert left and right output sound signals generated and input inside the information terminal 1 into left and right output sounds, respectively, and emit the sounds toward the user. When a user listens to sound through headphones, they may hear air-conducted sound, which enters the ear and is transmitted by air vibrations, or they may hear bone-conducted sound, which is transmitted by bone vibrations without passing through the ears. The headphones may be either air-conducted or bone-conducted, in other words, bone-conducted.

[0030] The headphones may be semi-closed headphones that are worn in contact with the surface of the ear, or open headphones that do not completely block the ear, i.e., open-ear headphones. When using open-ear headphones, the user can hear ambient sounds that enter through the headphones in addition to the original sound. In particular, among bone conduction headphones, open-ear headphones that do not block the ear at all are often used. Note that the audio output unit 14 is not limited to headphones, and may also be earphones or the like.

[0031] The communication unit 15 transmits and receives images of the virtual conference room, audio signals of spoken voices, and the like to and from a web conference server described below. The communication unit 15 includes various communication interfaces that handle processing based on multiple communication protocols, such as LAN and mobile communication. The communication unit 15 selects a communication protocol suitable for the usage environment from among the multiple communication protocols. The communication unit 15 also transmits and receives call data and the like via wireless communication with a base station in a mobile wireless communication network (not shown), and further transmits and receives various data via a core network.

[0032] For communication with base stations, etc., W-CDMA (Wideband Code Division Multiple Access, registered trademark) system, GSM (Global System for Mobile communications, registered trademark) system, LTE (Long Term Evolution) system, 5G (5th Generation) system, etc. are used. However, of course, other communication systems may also be used.

[0033] 1 includes components that are not essential to the embodiment. The information terminal 1 may not be provided with these components, or may further include other components depending on the application.

[0034] <Outline of Web Conference System> Figure 2 is a schematic diagram showing an example of the configuration of a web conference system according to one embodiment. The web conference system shown in Figure 2 includes multiple information terminals 6A-6D and a web conference server 2, which are connected to each other via a network 3. Information terminal 6A is installed at base 4A and used by participant 5A. Information terminal 6B is installed at base 4B and used by participant 5B. Information terminals 6C and 6D are installed at base 4C and used by participants 5C and 5D, respectively. In this example, four participants 5A-5D from three bases are participating in the web conference, but the number of bases and the number of participants can be changed as appropriate.

[0035] For example, information terminal 6A is a smartphone, information terminal 6B is an HMD, and information terminals 6C and 6D are PCs. Each of information terminals 6A to 6D corresponds to information terminal 1 shown in FIG. 1. Web conference server 2 includes a web conference service 2a, specifically, a storage device such as a storage device that stores programs and data required for the service, and a processor that executes the programs. Web conference server 2 provides web conference services to information terminals 6A to 6D. Note that the web conference server 2 may be configured with multiple servers by dividing functions, balancing loads, etc.

[0036] Web conference participants 5A-5D access the web conference server 2 via the network 3 using their respective information terminals 6A-6D and receive the web conference service 2a. For example, the avatar image and spoken voice of participant 5A are transmitted from information terminal 6A to the web conference server 2. The web conference server 2 receives the avatar images and spoken voice of all participants connected to the web conference service 2a and distributes them to the information terminals of each participant as the distributed images and spoken voice of the web conference service 2a. For example, the avatar image and spoken voice of participant 5A are transmitted to information terminals 6B-6D at other locations as the distributed images and spoken voice of the web conference service 2a.

[0037] Fig. 3 is a sequence diagram showing an example of the processing performed between the web conference application 21 of the information terminal 1 shown in Fig. 1 and the web conference service 2a of the web conference server 2 shown in Fig. 2. Explaining the sequence diagram from the viewpoint of the web conference application 21 of the information terminal 1, the information terminal 1 first logs in to the web conference service 2a in step S10.

[0038] Next, in step S11, the information terminal 1 receives an image of the virtual conference room where the web conference will be held, in other words, the virtual space. Next, in step S12, the information terminal 1 transmits an avatar image of the user of the information terminal 1 and position information of the avatar within the virtual conference room. The avatar position information includes information on the position and orientation of the avatar placed within the virtual conference room space. When transmitting the avatar image, the information terminal 1 may transmit the image data itself, or may select the avatar image from among multiple images provided by the web conference service 2a.

[0039] Next, in step S13, the information terminal 1 receives a list of other participants, and in step S14, receives an updated image of the virtual conference room in which avatar images of all participants are arranged. Thereafter, in step S15, the information terminal 1 receives a notification of the start of the conference. When the web conference starts, the processes of steps S16 and S17 are performed. In step S16, the information terminal 1 receives the speech of the other participants, and in step S17, it transmits the speech of the user of the information terminal 1, who is one of the participants.

[0040] The processes of steps S16 and S17 are repeated while the web conference continues. At this time, the processes of steps S16 and S17 are executed in any order. That is, the order of the processes of steps S16 and S17 may be reversed, or one of the processes of steps S16 and S17 may be repeated multiple times in succession. After such a web conference, when the information terminal 1 is notified of the end of the conference by the web conference service 2a in step S18, it logs out in step S19.

[0041] 4 is a diagram showing an example of the contents of the participant list 23 in FIGS. 1 and 3. The participant list 23 represents a list of participants participating in a web conference. The participant list 23 shown in FIG. 4 includes, for example, the following fields: participant information T10, participant display name T11, avatar image information T12, participant position T13, and participant orientation T14. Information on all participants is registered in each field, and information on participants who have left the virtual conference room is deleted.

[0042] The detailed contents of each item in the participant list 23 are as follows. "Participant information T10": Identification information for identifying each participant, assigned when participants 5A to 5D enter the virtual conference room in step S12, etc. "Participant display name T11": The name or nickname entered by each participant when they enter the virtual conference room in step S12, which is displayed near their avatar. "Avatar image information T12": The number of the avatar image selected by each participant in step S12. "Participant position T13": Information indicating the seat position and absolute position of each participant in the virtual conference room. "Participant orientation T14": Information indicating the direction in which each participant is facing in the virtual conference room.

[0043] Four participants 5A to 5D are registered in the participant list 23 shown in Fig. 4. As shown in Fig. 2, the four participants 5A to 5D are users of the information terminals 6A to 6D. Each participant specifies the seat position and the direction of their face after sitting down, for example, based on the image of the virtual conference room displayed on the information terminal 1 in step S11. This determines information on the participant position T13 and participant direction T14.

[0044] 5 is a diagram showing an example of the updated image of the virtual conference room in step S14 in FIG. 3, and is a diagram showing the positions of all participants in the virtual conference room. As shown in FIG. 5, in the virtual conference room 7, avatar images 7A-7D of all participants 5A-5D are arranged at designated positions and facing designated directions based on the participant list 23. In other words, the updated image is an image of the virtual space in which the avatar images of all participants are superimposed at positions based on the position information.

[0045] (First embodiment) <Details of web conference application> Figure 6 is a flowchart showing an example of processing content based on the web conference application 21 of the information terminal 1 shown in Figure 1 in the first embodiment. The flowchart shown in Figure 6 is executed by the information terminal 1 including the main processing unit 16 and the memory unit 17, in other words, by a computer system. In detail, the processor constituting the main processing unit 16 executes the flowchart using the web conference application 21 stored in the memory unit 17, which is one of the storage devices.

[0046] 6, in step S20, the information terminal 1 starts processing based on the web conference application 21. In step S21, the information terminal 1 logs in to the web conference service 2a provided by the web conference server 2 and participates in the web conference. In response, in step S22, the information terminal 1 receives an image of the virtual conference room from the web conference server 2 and displays it on the image display unit 11.

[0047] Next, in step S23, the information terminal 1 transmits to the web conference server 2 an avatar image of the user of the information terminal 1 and position information including the orientation of the avatar in the virtual conference room, as described in Figures 4 and 5. In response, in step S24, the information terminal 1 receives from the web conference server 2 a web conference participant list 23 and an updated image of the virtual conference room in which the avatars of all participants are arranged, as described in Figures 4 and 5. In particular, after all participants have adjusted their seat positions, the information terminal 1 receives an updated image that reflects the adjusted seat positions.

[0048] Next, in step S25, information terminal 1 receives a notification of the start of the web conference from web conference server 2 and starts the timeline. Furthermore, in step S26, information terminal 1 opens transcription file 25 or creates a new one if it has not been created yet, and records the start time of the conference, information included in participant list 23, etc. in transcription file 25.

[0049] In step S26, as described in FIG. 4, the participant list 23 includes, as position information, information such as the absolute position and orientation of each participant within the virtual conference room. Here, for example, the information terminal 6A used by participant 5A converts the absolute position and orientation information of the other participants 5B-5D included in the participant list 23 into information on relative positions and orientations with respect to participant 5A. The same is true for the other information terminals 6B-6D. In other words, the absolute position and orientation information of the other participants included in the participant list 23 is converted into information on relative positions and orientations with respect to each of participants 5B-5D. This makes it possible to easily generate stereophonic signals at each of the information terminals 6A-6D, as will be described in detail below.

[0050] Thereafter, while the conference continues, the information terminal 1 repeats the processes of steps S27 to S32. In step S27, the information terminal 1 receives the speech of the other participants, specifically, the speech signal, via the web conference server 2. In step S28, the information terminal 1 converts the received speech signal into a stereophonic signal and outputs it to the audio output unit 14. In step S29, the information terminal 1 converts the received speech into text and adds the text data to the transcription file 25 together with a timestamp.

[0051] Meanwhile, in step S30, information terminal 1 transmits the speech of the user of information terminal 1, specifically the speech signal, to the other information terminals via web conference server 2. In step S31, information terminal 1 converts the user's speech into text and adds it to transcription file 25 along with speaker information and a timestamp. In step S32, information terminal 1 determines whether a notification of the end of the conference has been received from web conference server 2, and if not ("NO"), returns to step S27 to continue the conference.

[0052] The order of steps S27 to S29 and steps S30 and S31 may be arbitrary, and either step may be repeated multiple times. Furthermore, while the conference is ongoing, the information terminal 1 may display the updated image of the virtual conference room received in step S24, i.e., the image shown in FIG. 5, on the image display unit 11.

[0053] If the information terminal 1 receives the notification of the end of the conference in step S32 (if "YES"), in step S33 the information terminal 1 appends the conference end time to the transcription file 25 and then closes it. Thereafter, in step S34 the information terminal 1 transmits a logout to the web conference server 2, and in step S35 the processing based on the web conference application 21 ends.

[0054] In the above description, in step S26, the information terminal 1 opens the transcription file 25 or creates a new one if one has not been created, and records the conference start time, information included in the participant list 23, and the like in the transcription file 25. In step S29, the information terminal 1 also converts the received speech into text and adds the text data together with a timestamp to the transcription file 25, and in step S31 converts the user's speech into text and adds the text data together with the speaker information and a timestamp to the transcription file 25. However, these processes may be executed not only by the information terminal 1 but also by the web conference server 2. This reduces the processing load on the information terminal 1.

[0055] <Details of the Transcription File> Fig. 7 is a diagram showing an example of the structure of the transcription file 25 generated by the flowchart shown in Fig. 6. As shown in Fig. 7, the transcription file 25 records metadata such as a participant list 30 in addition to text data of speech uttered in the web conference. These data are given meaning by markup tags.

[0056] The transcription file 25 begins with the conference name, followed by the start time. The start time is, for example, in a common standard time such as UTC. Following the start time is a participant list 30. The participant list 30 is a list for each participant, and includes the name, avatar information, position, and orientation data of each participant. Unlike the participant list 23 shown in FIG. 4, the participant list 30 in the transcription file 25 has its position and orientation converted to a relative position and orientation based on participant A.

[0057] Following the participant list 30, the contents of the web conference are recorded. Specifically, the contents of the web conference include a set of data sets 31 for each participant's utterance, which are repeatedly recorded until the end of the web conference. Each data set 31 includes, as components, a specific time on the timeline, i.e., a timestamp indicating the elapsed time of the conference, speaker information such as the speaker's name, and text data of the spoken voice. Each component is separated by a markup tag. Finally, the end time of the web conference is recorded, and the transcription file 25 is closed.

[0058] Figure 8 is a flowchart showing an example of the processing content when playing back the transcription file 25 shown in Figure 7. The flowchart shown in Figure 8 is also executed by a computer system, similar to the case of Figure 6. That is, the processor constituting the main processing unit 16 executes the flowchart using the web conference application 21 stored in the memory unit 17, which is one of the storage devices.

[0059] For example, if a user wishes to replay the contents of a web conference after it has ended, the user launches the web conference application 21 and causes the information terminal 1 to execute the flowchart. In Fig. 8, the information terminal 1 starts a playback process based on the web conference application 21 in step S40, and reads the transcription file 25 in step S41. Subsequently, the information terminal 1 extracts the position information, i.e., the position and orientation, included in the participant list 30 in step S42.

[0060] Next, in step S43, the information terminal 1 extracts text data of the spoken voice from the transcription file 25 and generates a voice signal of the spoken voice by voice synthesis. Furthermore, in step S44, the information terminal 1 converts the generated voice signal of the spoken voice into a stereophonic signal based on the position information in the participant list 30. Then, while referring to the time stamp, the information terminal 1 outputs the stereophonic signal to the audio output unit 14 so that the time intervals of the spoken voice are equivalent to those of an actual web conference.

[0061] In step S45, the information terminal 1 determines whether or not any speech data set 31 remains. Then, the information terminal 1 repeats the processes of steps S43 and S44 as long as any speech data set 31 remains in the transcription file 25 (step S45: NO). On the other hand, when the information terminal 1 runs out of speech data set 31 (step S45: YES), in step S46, the information terminal 1 ends the playback process based on the web conference application 21.

[0062] The information terminal 1 may associate the updated image of the virtual conference room received in step S24 of FIG. 6 with the transcription file 25 and store it in the storage unit 18. In this case, the information terminal 1 may display the updated image of the virtual conference room on the image display unit 11 while playing the transcription file 25 in FIG. 8. Furthermore, when synthesizing the voice in step S43, for example, data for multiple voice qualities may be provided in advance, and the user may select an easy-to-listen voice quality from the multiple voice qualities for each participant. In this case, the web conference application 21 may be provided with a user interface that allows each participant to select one of the multiple voice qualities. This selection can be made at any time while the transcription file 25 is being played.

[0063] In the above description, the transcription file 25 was played back by the web conference application 21. However, the transcription file 25 may be played back not only by the web conference application 21 but also by another application or a function provided in the OS of the information terminal 1.

[0064] <Regarding Stereophonic Signals> Fig. 9 is a flowchart showing an example of detailed processing involved in outputting stereophonic signals in Fig. 6 or Fig. 8. This flowchart is executed in step S28 in Fig. 6 or step S44 in Fig. 8. As shown in Fig. 9, the information terminal 1 determines a head-related transfer function for each participant in step S50.

[0065] In step S50, the information terminal 1, in step S51, refers to the relative position and relative orientation of each of the other participants relative to the user of the information terminal 1, which are included in the participant list 30. Then, in step S52, the information terminal 1 determines the head-related transfer function for each of the other participants by referring to the head-related transfer function database 24 using the relative position and relative orientation of each of the other participants.

[0066] Specifically, for example, sound transfer functions from sound sources arranged in various positions and directions around the head to both ears of the head are registered in advance in the head transfer function database 24. The information terminal 1 determines the head transfer function for each of the other participants by referring to the head transfer function database 24, regarding the relative positions and relative orientations of the other participants as the positions and directions of the sound sources.

[0067] Next, in step S53, the information terminal 1 generates a stereophonic signal by performing a convolution operation between the speech signal and the determined head-related transfer function, or an equivalent operation. That is, the information terminal 1 converts the input speech signal into a stereophonic signal. Then, in step S54, the information terminal 1 outputs the generated stereophonic signal to the audio output unit 14 shown in FIG. 1. The speech input in step S53 is the speech received from the web conference server 2 in the case of step S28, or is the speech converted by speech synthesis from text data extracted from the transcription file 25 in the case of step S44.

[0068] <Major Effects of the First Embodiment> As described above, the method of the first embodiment allows stereophonic signals to be applied to a web conference by determining the positions of participants in the virtual space where the web conference is held, and further allows a web conference to be reproduced using text-to-speech conversion. This makes it possible, for example, to improve the convenience of web conference users and realize and reproduce high-quality web conferences, i.e., web conferences that are close to face-to-face meetings. In particular, the ability to reproduce a web conference using stereophonic signals makes it possible to more faithfully reproduce a review of the web conference even after it has ended.

[0069] (Second embodiment) <Details of web conference application> For example, a participant in a web conference may want to change his or her position or orientation in the virtual conference room to make it easier to hear the speech of a specific participant during the web conference. In the second embodiment, a method for enabling such a change operation will be described.

[0070] Fig. 10 is a diagram showing an example of an image of a virtual conference room obtained by modifying Fig. 5, and similar to Fig. 5, it is an image showing the arrangement of participants in the virtual conference room. In Fig. 10, participant avatar images 7A to 7D have already been arranged in the virtual conference room 7, similar to Fig. 5, but the position of participant A is different from that in Fig. 5. That is, participant A is positioned as shown in Fig. 5 at the start of the web conference, but is changed to the position shown in Fig. 10 during the web conference.

[0071] Fig. 11 is a flowchart showing an example of processing based on the web conference application 21 of the information terminal 1 shown in Fig. 1 in the second embodiment. The flowchart shown in Fig. 11 is also executed by a computer system, similar to the case of Fig. 6. That is, the processor constituting the main processing unit 16 executes the flowchart using the web conference application 21 stored in the memory unit 17, which is one of the storage devices.

[0072] 11 differs from the flowchart shown in FIG. 6 in that processing related to rearrangement in steps S60 to S63 is added between steps S26 and S27. In step S60, the information terminal 1 determines whether the user of the information terminal 1 has performed an operation to rearrange the information terminal 1. If an operation to rearrange the information terminal 1 has not been performed in step S60 (if "NO"), the information terminal 1 proceeds to step S27 and receives the spoken voice.

[0073] On the other hand, if a position change operation has been performed in step S60 (if "YES"), the information terminal 1 transmits position information including the changed position and orientation to the web conference server 2 in step S61. In response, the information terminal 1 receives the participant list 23 reflecting the position change and an updated image of the virtual conference room from the web conference server 2 in step S62. The information terminal 1 updates the display image of the information terminal 1 with the received image of the virtual conference room. Then, in step S63, the information terminal 1 converts the changed position and orientation included in the received participant list 23 into relative values ​​and adds them to the transcription file 25, and then proceeds to step S27.

[0074] In the flowchart of the processing, an example in which the user of the information terminal 1 performs a position change operation has been described. On the other hand, for example, there may be cases in which it is desirable for a participant who is making a presentation during a web conference and sharing materials with all participants to be repositioned so that he or she is positioned directly in front of participants who are not making a presentation. In this case, for example, the web conference application may make a determination at the time when the operation to share materials with all participants is performed during the web conference, and transmit the participant position change to the web conference server 2. Then, the processing from step S62 onward may be performed.

[0075] <Details of Transcription File> Figure 12 is a diagram showing an example of the configuration of a transcription file 25 generated by the flowchart shown in Figure 11. In the transcription file 25 shown in Figure 12, the chronological content of the transcription file 25 shown in Figure 7 is temporarily interrupted due to the processing in step S63 in Figure 11. Then, at the interrupted position, more specifically, at a position between data sets 31 in the transcription file 25 that indicates the time point at which the rearrangement occurred, a participant list 30a reflecting the rearrangement is added. After that, the chronological content resumes.

[0076] The layout change may also be made by another participant in the web conference. In this case, the information terminal 1 receives the participant list and updated image of the virtual conference room associated with step S62 from the web conference server 2, for example, by interrupt processing, and then performs the processing in step S63. As a result, even if the layout change is made during the web conference, the web conference can continue with the changed layout shared by all participants.

[0077] The position change may also be applied when playing back the transcription file 25, as described in FIG. 8 . This allows, for example, creating a situation in which the speech of a specific participant is easy to hear when checking minutes, etc. A situation in which the speech of a specific participant is easy to hear means, for example, placing the speech of the specific participant directly in front of the user performing the playback. This allows the user performing the playback to hear the speech from directly in front of them, making it easier to hear the speech than when the speech is heard from a position other than directly in front.

[0078] The layout change can be realized, for example, by providing a user interface that allows a participant to be selected in the web conference application 21 and the layout of the selected participant to be changed. This layout change can be made at any time while the transcription file 25 is being played. If the layout change is applied when the transcription file 25 is being played, the transcription file 25 is not rewritten. In other words, after the web conference ends, each participant does not need to share their layout with other participants, and can freely determine the layout of all participants.

[0079] <Major Effects of the Second Embodiment> As described above, the method of the second embodiment can also achieve the same effects as those described in the first embodiment. Furthermore, since the location of participants can be changed during or after a web conference, it is possible to further improve user convenience and the quality of the web conference, for example. In particular, by being able to change the location of participants when replaying a web conference, it is possible to replay a web conference after the end of the web conference in a situation where the speech of a participant that was difficult to hear during the web conference can be easily heard.

[0080] (Third embodiment) <Details of web conference application> In the third embodiment, a method is described in which characteristics of spoken voice are extracted during a web conference, and the extracted characteristics of spoken voice are reflected when generating spoken voice by speech synthesis from text data of the spoken voice after the web conference.

[0081] Fig. 13 is a flowchart showing an example of processing based on the web conference application 21 of the information terminal 1 shown in Fig. 1 in the third embodiment. The flowchart shown in Fig. 13 is also executed by a computer system, similar to the case of Fig. 6. That is, the processor constituting the main processing unit 16 executes the flowchart using the web conference application 21 stored in the memory unit 17, which is one of the storage devices.

[0082] 13 differs from the flowchart shown in FIG. 6 in that step S70 is added between step S27 and step S28. After receiving the speech in step S27, the information terminal 1 extracts features of the speech of the other participants in step S70. Although not shown in the figure, the information terminal 1 may similarly extract features of the speech of the user of the information terminal 1 after transmitting the speech in step S30.

[0083] Examples of speech characteristics include the high-low distribution of frequency components contained in the speech, the pitch, the speed, and the dynamics of the speech. These speech characteristics are used when synthesizing speech from text data in step S43 during playback of the transcription file 25 shown in FIG. 8. In other words, when synthesizing speech, the pitch, speed, dynamics, etc. are tuned for the sound source based on the extracted speech characteristics. This allows a user, for example, when checking the minutes of a meeting, to experience an atmosphere closer to that of an actual web conference.

[0084] As a method for synthesizing speech that reflects the characteristics of speech, for example, there is a method in which multiple sound sources with different sound qualities are prepared in advance in the information terminal 1. In this case, the sound source with the closest characteristics can be selected from the multiple sound sources based on the extracted characteristics of the speech. Furthermore, the pitch, speed, strength, etc. of the selected sound source can be tuned based on the extracted characteristics of the speech. This makes it possible to generate speech that sufficiently reflects the voice quality of each participant to a certain extent.

[0085] Another method for performing voice synthesis is to use AI (artificial intelligence) for learning. In this case, for example, the AI ​​is trained on the speech of each participant throughout the entire web conference to determine the AI ​​parameters required to obtain the voice quality of each participant. These AI parameters are recorded in the transcription file 25. When the transcription file 25 is played back, the recorded AI parameters are set in the AI ​​model to perform voice synthesis. Note that, from the perspective of the processing load associated with learning, the web conference server 2 may perform AI learning and then transmit the AI ​​parameters resulting from the learning to the information terminal 1.

[0086] <Details of Transcription File> Figure 14 is a diagram showing an example of the configuration of the transcription file 25 generated by the flowchart shown in Figure 13. Unlike the transcription file 25 shown in Figure 7, the transcription file 25 shown in Figure 14 has speech features 35 added to a set of data 31 for each participant's utterance. That is, each data set 31 includes the speech features 35 as components in addition to the timestamp, speaker name, and text data of the speech.

[0087] <Major Effects of the Third Embodiment> As described above, the method of the third embodiment can also achieve the same effects as those described in the first embodiment. Furthermore, when playing back the transcription file 25, it is possible to allow the user to experience an atmosphere close to that of an actual web conference. In other words, it is possible to more faithfully reproduce an actual web conference. In other words, it is possible to reproduce the detailed nuances of each participant's remarks in the web conference.

[0088] (Fourth embodiment) <Details of web conference application> In the fourth embodiment, a method is described in which waveform data of spoken voice is saved during a web conference, and the saved waveform data of spoken voice is reflected when generating spoken voice from text data of the spoken voice after the web conference.

[0089] Fig. 15 is a flowchart showing an example of processing based on the web conference application 21 of the information terminal 1 shown in Fig. 1 in the fourth embodiment. The flowchart shown in Fig. 14 is also executed by a computer system, similar to the case of Fig. 6. That is, the processor constituting the main processing unit 16 executes the flowchart using the web conference application 21 stored in the memory unit 17, which is one of the storage devices.

[0090] The flowchart shown in FIG. 15 differs from the flowchart shown in FIG. 6 in that step S71 is added between step S27 and step S28. After receiving the speech in step S27, the information terminal 1 acquires waveform data of the speech of the other participants in step S71 and records the waveform data in the transcription file 25. Although not shown, the information terminal 1 may similarly acquire and record waveform data of the speech of the user of the information terminal 1 after transmitting the speech in step S30. The waveform data of the speech of the participants may be in an audio file format. For example, an existing format such as WAV, AIFF, AAC, mp3, or mp4 / m4a may be used.

[0091] These waveform data of the speech are used to synthesize speech from text data in step S43 when reproducing the transcription file 25 shown in Fig. 8. This allows the user, for example, when checking the minutes of a meeting, to experience a sense of realism equivalent to that of an actual web conference.

[0092] <Details of Transcription File> Figure 16 is a diagram showing an example of the configuration of the transcription file 25 generated by the flowchart shown in Figure 15. Unlike the transcription file 25 shown in Figure 7, the transcription file 25 shown in Figure 16 has speech waveform data 36 added to a set of data 31 for each participant's utterance. That is, each data set 31 includes, as components, the timestamp, the speaker's name, text data of the speech, and the speech waveform data 36.

[0093] <Major Effects of the Fourth Embodiment> As described above, the method of the fourth embodiment can also achieve the same effects as those described in the first embodiment. Furthermore, when playing back the transcription file 25, it becomes possible for the user to experience a sense of realism equivalent to that of an actual web conference. In other words, it becomes possible to more faithfully reproduce an actual web conference.

[0094] Fifth Embodiment <Details of Web Conference Application> In the fifth embodiment, a method for editing the transcription file 25 by adding a playback control command to the transcription file 25 will be described.

[0095] Fig. 17 is a flowchart showing an example of processing based on the web conference application 21 of the information terminal 1 shown in Fig. 1 in the fifth embodiment. The flowchart shown in Fig. 17 is also executed by a computer system, similar to the case of Fig. 6. That is, the processor constituting the main processing unit 16 executes the flowchart using the web conference application 21 stored in the memory unit 17, which is one of the storage devices.

[0096] 17, in step S80, the information terminal 1 starts editing the transcription file 25 based on the web conference application 21. First, in step S81, the information terminal 1 reads the transcription file 25. Next, in step S82, the information terminal 1 starts playing back the transcription file 25.

[0097] Next, the information terminal 1 pauses playback in response to a command from the editor at a point where a playback control command should be inserted into the transcription file 25. If playback is paused in step S83 (if "YES"), the information terminal 1 selects a playback control command to insert in step S84 based on the editor's command. Then, in step S85, the information terminal 1 adds the selected playback control command to the transcription file 25. In this case, for example, the web conference application 21 may be provided with a user interface that allows the user to select one of multiple playback control commands. This selection can be made at any time while the transcription file 25 is being played back.

[0098] Next, in step S86, the information terminal 1 cancels the pause in response to a command from the editor and resumes playback of the transcription file 25. Thereafter, in step S87, the information terminal 1 determines whether playback of the transcription file 25 has finished, and if not (in the case of "NO"), returns to step S83 and repeats inserting the playback control command. On the other hand, if playback has finished (in the case of "YES") in step S87, the information terminal 1 closes the transcription file 25 in step S88 and ends the editing process in step S89.

[0099] In the flowchart of this process, the editor pauses and resumes playback to insert playback control commands more accurately. However, the editor can also insert playback control commands in real time without pausing and resuming playback.

[0100] <Details of Transcription File> Figure 18 is a diagram showing an example of the configuration of a transcription file 25 generated by the flowchart shown in Figure 17. The transcription file 25 shown in Figure 18 differs from the transcription file 25 shown in Figure 7 in that playback control commands 37 are added to a set of data sets 31 for each participant's utterance. That is, each data set 31 includes, as components, a timestamp, a speaker name, and text data of the uttered voice, as well as a playback control command 37 based on an editor's instructions. The playback control command 37 is a command that controls, for each data set 31, the method of playing back the uttered voice generated by speech synthesis.

[0101] 18, a command for specifying double-speed playback (FAST 2X), a command for specifying skipping (SKIP), and a command for specifying the number of repetitions (REPEAT Twice) are inserted as playback control commands 37. However, it is not necessary to add playback control commands 37 to all data sets 31. In other words, playback control commands 37 are not necessary for data sets 31 that perform normal playback. Furthermore, the playback control commands 37 are not limited to the commands described above, and it is sufficient if one of the playback control commands 37 is any of the commands described above, for example.

[0102] 17 and 18, the information terminal 1 inserted the playback control commands 37 while pausing and resuming for each data set 31. However, the method for inserting the playback control commands 37 may be changed as appropriate. For example, the playback control commands 37 may be inserted for each participant, such as always applying a double-speed playback command to participant D and always applying normal playback to participant A. In this case, the information terminal 1 may search the transcription file 25 by speaker name, for example, and insert the playback control commands 37 into the data sets 31 that are found.

[0103] <Major Effects of the Fifth Embodiment> Using the method of the fifth embodiment, the same effects as those described in the first embodiment can be achieved. Furthermore, the transcription file 25 can be played back for various purposes, such as shortening the review time or emphasizing important points by repeating them. This allows for efficient review of the contents of a meeting after it has ended.

[0104] Furthermore, the edited transcription file 25 can be distributed, for example, as summarized minutes of a meeting. As a specific example, the information terminal 1 determines that the text of a speech for which double-speed playback (FAST 2X) or skip (SKIP) is specified as the playback control command 37 is not important, and deletes the speaker's name and the text of the speech. On the other hand, the information terminal 1 determines that the text of a speech for which a repeat count (REPEAT Twice) is specified is important, and leaves the speaker's name and the text of the speech intact. For example, such processing can create summarized minutes of a meeting. As a result, user convenience can be further improved.

[0105] Sixth Embodiment <Details of Web Conference Application> In the sixth embodiment, a method will be described in which only text data of uttered voice is extracted from the transcription file 25 and a text file is output.

[0106] Fig. 19 is a flowchart showing an example of processing based on the web conference application 21 of the information terminal 1 shown in Fig. 1 in the sixth embodiment. The flowchart shown in Fig. 19 is also executed by a computer system, similar to the case of Fig. 6. That is, the processor constituting the main processing unit 16 executes the flowchart using the web conference application 21 stored in the memory unit 17, which is one of the storage devices.

[0107] 19, when the information terminal 1 starts the text data extraction process based on the web conference application 21 in step S90, it first generates a text file in step S91. Alternatively, the information terminal 1 opens an already generated text file. Next, the information terminal 1 reads the transcription file 25 in step S92, and starts playing the transcription file 25 in step S93.

[0108] Next, in step S94, the information terminal 1 determines whether the markup tag is text, and if it is text (if "YES"), in step S95, the information terminal 1 extracts the text data and writes it to a text file. Thereafter, in step S96, the information terminal 1 determines whether playback of the transcription file 25 has finished, and if it has not finished (if "NO"), the information terminal 1 returns to step S94 and repeats the extraction of text data. On the other hand, if playback has finished in step S96 (if "YES"), the information terminal 1 closes the text file in step S97 and ends the extraction process in step S98. Note that the extraction of text data in step S95 may extract both the speaker name and the text of the spoken voice, or may extract only one of them.

[0109] <Major Effects of the Sixth Embodiment> As described above, the method of the sixth embodiment can also achieve the same effects as those described in the first embodiment. Furthermore, by generating a text file from the transcription file 25, the text file can be used in an environment that does not include the web conference application 21. For example, even in a program other than the web conference application 21, the text file can be read aloud. As a result, user convenience can be further improved.

[0110] An information terminal according to one embodiment includes a computer system including a CPU and a memory, a communication unit, a storage unit, an audio input unit, an audio output unit, an image display unit, an operation input unit, and a camera / sensor unit. The storage unit stores a program for a web conference application executed by the computer system.

[0111] A user of an information terminal runs a web conference application and receives web conference services from an external server via a communication unit. The user's speech is transmitted to the web conference service via the voice input unit and communication unit and distributed to other participants in the web conference. The speech of the other participants is amplified from the user's information terminal via the communication unit and voice output unit.

[0112] Furthermore, the user of the information terminal registers their location information in the virtual space of the conference room, which is provided by the web conference service and displayed on the image display unit. Other participants also register their location information in the virtual space in the same way. This allows the user and other participants to share their positional relationships in the virtual space. The web conference application uses the positional relationships to amplify the speech of other participants as stereophonic signals from the user's information terminal.

[0113] Furthermore, the speech of the web conference participants is converted into text data through a transcription process. At this time, the text data of the speech separated by markup tags, speaker information, and timestamps of the conference elapsed time are compiled into a single data set, and a transcription file recording each data set of speech from each participant is output to the storage unit.

[0114] Furthermore, the text data obtained through transcription processing can be used to reconstruct a stereophonic signal using the positional relationship, allowing users to hear the speech of the web conference with a sense of realism even after the web conference has ended.

[0115] In one embodiment of the audio processing method for an information terminal, a stereophonic signal is generated and output from the positional relationship between the user of the information terminal and other participants and the speech of the web conference obtained from the web conference service. Furthermore, a transcription file is output to a storage unit, recording a data set for each participant's speech, with the text data of the speech separated by markup tags, speaker information, and a timestamp of the conference elapsed time as a set of data. The transcription file output to the storage unit may be configured to be output to another information terminal via a communication unit of the information terminal and to be shared with other information terminals. This provides the effect of, for example, enabling even those who were unable to participate in the web conference to understand the content of the web conference.

[0116] Although the embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above-described embodiments, and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. Furthermore, numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0117] For example, in the above-described embodiment, an example has been described in which the information terminal 1 generates the transcription file 25. However, instead of the information terminal 1, the web conference server 2 may generate the transcription file 25 and distribute it to the information terminal 1. Alternatively, the information terminal 1 and the web conference server 2 may cooperate to generate the transcription file 25. In this case, the information terminal 1 may, for example, cause the web conference server 2 to perform speech recognition or the like and generate the transcription file 25 while obtaining the results.

[0118] The programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each process is performed may also be changed.

[0119] Some or all of the functions constituting the above-described embodiments may be realized in hardware, for example, by designing them as integrated circuits. Alternatively, they may be realized in software by a microprocessor unit, CPU, or the like interpreting and executing an operating program that realizes each function. Alternatively, some of the functions may be implemented in software, thereby using both hardware and software. Alternatively, some or all of the functions may be realized in a server.

[0120] The server may take any form as long as it can execute functions in cooperation with other components via communication, and may be, for example, a local server, a cloud server, an edge server, an internet service, etc. Information such as programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD, or may be stored in a device on a communication network.

[0121] Furthermore, the control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all of the control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected.

[0122] 1, 6A to 6D: Information terminal, 2: Web conference server, 2a: Web conference service, 3: Network, 4A to 4C: Base, 5A to 5D: Participant, 7: Virtual conference room, 7A to 7D: Avatar image, 10: Camera / sensor unit, 11: Image display unit, 12: Operation input unit, 13: Audio input unit, 14: Audio output unit, 15: Communication unit, 16: Main processing unit, 17: Memory unit, 18: Storage unit, 19: Internal bus, 20: Basic operation program, 21: Web conference application, 22: Data, 23, 30, 30a: Participant list, 24: Head-related transfer function database, 25: Transcription file, 31: Data set, 35: Audio characteristics, 36: Waveform data, 37: Playback control command

Claims

1. An information terminal comprising: a storage device storing a program for participating in a WEB conference, a participant list representing information of participants in the WEB conference, and a database of head-related transfer functions for generating a stereophonic signal; a processor for executing the program; a display displaying an image of a virtual space in which the WEB conference is held; and an audio output interface connected to a sound emitting device, wherein the participant list includes position information of the participants in the virtual space, and the processor determines the head-related transfer functions of the participants by referring to the database using the position information included in the participant list, and based on the determined head-related transfer functions, converts speech sounds in the WEB conference into stereophonic signals and outputs them to the audio output interface; and generates a transcription file in which the dataset for each utterance of the participant is written based on the speech sounds in the WEB conference, using a dataset whose components are text data of the speech sounds separated by markup tags, speaker information, and a timestamp of the elapsed time of the conference, and stores the transcription file in the storage device.

2. An information terminal according to claim 1, wherein the processor causes the display to display an image of the virtual space in which avatar images of the participants are superimposed at positions based on the position information.

3. An information terminal as described in claim 1, wherein when playing back the transcription file, the processor generates speech by speech synthesis from the text data of the speech, determines the head-related transfer function of the participant by referring to the database using the position information included in the participant list, and converts the speech generated by speech synthesis into a stereophonic signal based on the determined head-related transfer function, and outputs it to the audio output interface.

4. An information terminal as described in claim 3, wherein the processor records playback control commands based on instructions from a user of the information terminal as components of the data set in the transcription file, the playback control commands being commands that control a method of playing back the speech generated by the speech synthesis for each data set, and one of the playback control commands being a command that specifies the playback speed, or a command that specifies a skip, or a command that specifies the number of repetitions.

5. An information terminal according to claim 1, wherein the location information in the participant list includes information on the relative locations of the other participants based on the location of a user of the information terminal who is one of the participants.

6. An information terminal as described in claim 1, wherein the processor records the participant list including the changed location information in response to a change in the location information that occurs during the WEB conference, at a position between the data sets in the transcription file that represents the time when the change in the location information occurred.

7. An information terminal according to claim 1, wherein said processor extracts characteristics of speech uttered in said WEB conference and records the extracted characteristics of speech uttered in said transcription file.

8. An information terminal as described in claim 1, wherein the processor acquires waveform data of the speech in the WEB conference and records the acquired waveform data of the speech as a component of the data set in the transcription file.

9. An information terminal according to claim 1, wherein the processor extracts text data of the spoken voice from the transcription file, writes the extracted text data into a text file, and stores the text file in the storage device.

10. A voice processing method for an information terminal for participating in a WEB conference, comprising the steps of: storing a participant list representing information of participants in the WEB conference and a database of head-related transfer functions for generating a stereophonic signal; displaying an image of a virtual space in which the WEB conference is held; determining the head-related transfer functions of the participants by referring to the database using position information of the participants in the virtual space included in the participant list, and converting and outputting speech sounds in the WEB conference into stereophonic signals based on the determined head-related transfer functions; generating a transcription file in which the data set is recorded for each utterance of the participants based on the speech sounds in the WEB conference, using a data set whose components are text data of the speech sounds separated by markup tags, speaker information, and a timestamp of the elapsed time of the conference, and storing the transcription file.

11. A voice processing method for an information terminal as described in claim 10, wherein, when playing back the stored transcription file, a speech is generated by voice synthesis from the text data of the speech, the position information included in the participant list is used to refer to the database to determine the head-related transfer function of the participant, and the speech generated by voice synthesis is converted into a stereophonic signal based on the determined head-related transfer function and output.

12. A conference system comprising: a plurality of information terminals connected to one another via a network; and a web conference server connected to the network and providing a web conference service to the plurality of information terminals, wherein each of the plurality of information terminals comprises: a storage device for storing a program for participating in the web conference and a database of head-related transfer functions for generating a stereophonic signal; a processor for executing the program; a display for displaying an image of a virtual space in which the web conference is held; and an audio output interface connected to a sound emitting device, wherein the processor receives a participant list including position information of participants in the web conference in the virtual space from the web conference server, stores the participant list in the storage device, determines the head-related transfer functions of the participants by referring to the database using the position information included in the participant list, converts speech sounds in the web conference into stereophonic signals based on the determined head-related transfer functions, and outputs the signals to the audio output interface, A conference system which uses a data set whose components are text data of spoken voice separated by markup tags, speaker information, and a timestamp of the conference elapsed time, generates a transcription file that records the data set for each utterance of the participants based on the spoken voice in the web conference, and stores the transcription file in the storage device.

13. A conference system according to claim 12, wherein the processor receives from the WEB conference server an image of the virtual space in which the avatar images of the participants are superimposed at positions based on the position information, and displays the image on the display.

14. A conference system as described in claim 12, wherein when playing back the transcription file, the processor generates speech by speech synthesis from the text data of the speech, determines the head-related transfer function of the participant by referring to the database using the position information included in the participant list, and converts the speech generated by speech synthesis into a stereophonic signal based on the determined head-related transfer function, and outputs it to the audio output interface.

15. A conference system as described in claim 12, wherein the position information in the participant list received by the processor from the WEB conference server includes information on the absolute positions of the participants in the virtual space, and the processor converts the absolute position information included in the participant list into relative position information of the other participants based on the position of a user of one of the participants, the information terminal, and references the database using the relative position information.

Images