Multipoint conference control device, signal conversion device, terminal device, and programs therefor
The multipoint conference system addresses the challenge of incorporating vibration signals by using a control device and terminal devices with signal conversion, ensuring effective tactile conferencing without speaker damage and allowing user-selectable media presentation.
Patent Information
- Application Number
- JP2021128208
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-08-04
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2041-08-04
Smart Images

Figure 0007578239000001 
Figure 0007578239000002 
Figure 0007578239000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a conference system that realizes a conference including a haptic sense via communication. [Background technology]
[0002] Various systems have been put into practical use to realize a multi-point conference using audio and video via communication such as the Internet, and a representative one is the multi-point conference system described in Non-Patent Document 1. In a multi-point conference system such as that described in Non-Patent Document 1, audio signals, video signals, and screen information from each connected point are selected to generate audio signals, video signals, and screen information for playback, and the generated audio signals, video signals, and screen information are played back at each connected point, thereby realizing a multi-point conference via communication.
[0003] Non-Patent Document 2 discloses a technology for transmitting vibrations via communication in addition to conventionally transmitted sound and video. Non-Patent Document 2 discloses a technology for realizing a conference including tactile sensation via communication, in which a terminal device at each point connected to a communication line acquires sound, video, and vibrations at the point, respectively, and outputs them to the communication line as audio signals, video signals, and vibration signals, and reproduces sound, video, and vibrations from the audio signals, video signals, and vibration signals input from the communication line, respectively. Non-Patent Document 2 discloses that, in transmission via a communication line, one of two channels prepared for transmitting audio signals is used for transmitting the audio signals, and the other of the two channels prepared for transmitting the audio signals is used for transmitting the vibration signals. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] Cisco and / or its affiliates, "Cisco Webex", [online], [searched June 28, 2021], Internet<URL: https: / / www.webex.com / ja / all-new-webex.html> [Non-Patent Document 2] Hirohiko Hayakawa and 7 others, "Proposal of a bidirectional audiovisual and haptic medium "Public Haptic Legend" for remote communication with a high sense of reality," Transactions of the Virtual Reality Society of Japan Summary of the Invention [Problem to be solved by the invention]
[0005] In the multipoint conference system of Non-Patent Document 1, a conference at multiple points can be held using audio signals, video signals, and screen information, but the transmission of vibration signals for conveying tactile sensations is not taken into consideration, and a conference at multiple points including vibration signals cannot be held. If the technology of Non-Patent Document 2 is applied to the multipoint conference system of Non-Patent Document 1, a conference at multiple points including vibration signals can be held. However, in a conference at multiple points in which the technology of Non-Patent Document 2 is applied to the multipoint conference system of Non-Patent Document 1, one of the multiple channels that are originally for audio signals is used as a channel for vibration signals, so if a terminal device that normally reproduces audio signals from multiple channels is used, the terminal device will output the vibration signal of the channel for vibration signals to the speaker, which may cause problems such as the speaker emitting unexpected sounds, breaking down, or being destroyed. The present invention aims to realize a conference at multiple points in which the occurrence of problems during reproduction as described above is suppressed in a configuration in which one or more of the multiple channels that are originally for audio signals are used for vibration signals. [Means for solving the problem]
[0006] In order to solve the above-mentioned problems, in the present invention, in a multipoint conference system comprising a plurality of terminal devices and a multipoint conference control device connected to each terminal device via a communication path, and substituting one or more of a plurality of channels originally used for acoustic signals for vibration signals, a device for inputting a signal to the terminal devices converts the signal for terminal devices that cannot present a vibration signal or terminal devices that do not present a vibration signal so that the vibration signal is not included in the channel substituted for the vibration signal, and converts the signal for terminal devices that cannot present an acoustic signal or terminal devices that do not present an acoustic signal so that the acoustic signal is not included in the channel for acoustic signals that is not substituted for vibration signals.
[0007] Also, in the present invention, the terminal device displays on a display screen a presentation signal selection user interface area including a media group switching button which, when pressed, performs a selection by cyclic state switching including at least three states, a state in which sound is presented without presenting vibration, a state in which both sound and vibration are presented, and a state in which vibration is presented without presenting sound, and a media display group which, when one of the states is selected by pressing the media group switching button, displays so that it is visible whether the state is a state in which sound is presented without presenting vibration, a state in which both sound and vibration are presented, or a state in which vibration is presented. Effect of the Invention
[0008] According to the present invention, in a configuration in which one or more channels of a plurality of channels that are originally used for audio signals are used for vibration signals, it is possible to realize a conference at multiple points while suppressing the occurrence of problems during playback. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a block diagram illustrating an example of a multipoint conferencing system 10, 12. [Diagram 2] FIG. 2 is a block diagram showing an example of a terminal device 200-m (a tactile-enabled terminal device 200-m). [Diagram 3]FIG. 3 is a flow diagram showing an example of the processing of the transmitting device 210-m (haptically-enabled transmitting device 210-m). [Figure 4] FIG. 4 is a diagram showing an example of channels constituting a two-channel stereo and signals stored in each channel by the transmitting device 210-m (haptic-enabled transmitting device 210-m). [Diagram 5] FIG. 5 is a diagram showing an example of channels constituting 5.1 channel surround and signals stored in each channel by the transmitting device 210-m (haptic-enabled transmitting device 210-m). [Figure 6] FIG. 6 is a block diagram illustrating an example of a multipoint conference control device 100. As shown in FIG. [Figure 7] FIG. 7 is a flowchart showing an example of the processing of the multipoint conference control device 100. As shown in FIG. [Figure 8] FIG. 8 is a flowchart showing an example of the process of the conference signal generating unit 110. As shown in FIG. [Figure 9] FIG. 9 is a flow chart showing an example of the processing of the receiving device 220-m (haptically-enabled receiving device 220-m). [Figure 10] FIG. 10 is a block diagram showing an example of a multipoint conference system 11. As shown in FIG. [Figure 11] FIG. 11 is a block diagram showing an example of a tactile non-enabled terminal device 400-n. [Figure 12] FIG. 12 is a flow diagram illustrating an example of the processing of a non-haptically enabled sender device 410-n. [Figure 13] FIG. 13 is a block diagram illustrating an example of a multipoint conference control device 101. [Figure 14] FIG. 14 is a flowchart showing an example of the processing of the multipoint conference control unit 101. [Figure 15] FIG. 15 is a flowchart showing an example of the process of the conference signal generating unit 111. [Figure 16] FIG. 16 is a flow diagram illustrating an example of the processing of a non-haptically enabled receiver device 420-n. [Figure 17] FIG. 17 is a block diagram illustrating an example of a multipoint conference controller 102. [Figure 18]FIG. 18 is a flowchart showing an example of the processing of the multipoint conference control device 102 and the signal conversion device 604-m. [Figure 19] FIG. 19 is a flowchart showing an example of the process of the conference signal generating unit 112. [Figure 20] FIG. 20 is a diagram showing an example of a screen 2211-m of the video presenting section 221-m. [Figure 21] FIG. 21 is a diagram showing an example of a screen 2211-m of the video presenting section 221-m. [Figure 22] FIG. 22 is a block diagram showing an example of a multipoint conference system 13. As shown in FIG. [Figure 23] FIG. 23 is a block diagram illustrating an example of the multipoint conference controller 103. [Figure 24] FIG. 24 is a flowchart showing an example of the processing of the multipoint conference control unit 103. [Diagram 25] FIG. 25 is a block diagram showing an example of a signal conversion device 603-n. [Figure 26] FIG. 26 is a flow chart showing an example of the processing of the signal conversion device 603-n. [Figure 27] FIG. 27 is a flowchart showing an example of the process of the conference signal converter 613-n. [Figure 28] FIG. 28 is a block diagram illustrating an example of a multipoint conference system 14. [Figure 29] FIG. 29 is a block diagram illustrating an example of a signal conversion device 604-m. [Diagram 30] FIG. 30 is a diagram illustrating an example of the functional configuration of a computer that realizes each device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] <Reference form> Before describing the embodiments of the present invention, a multipoint conference system that is the premise of the present invention will be described as a reference embodiment. The multipoint conference system of the reference embodiment is a multipoint conference system of the non-patent document 1 to which the technology of non-patent document 2 is applied, and uses one or more of the multiple channels prepared for audio signals in an existing audio transmission method to transmit a signal representing vibration to be perceived as a touch (hereinafter referred to as a "vibration signal"), and uses a multipoint conference control device prepared for multipoint connection of video signals and audio signals to perform multipoint connection of vibration signals in addition to video signals and audio signals, thereby enabling the realization of a multipoint conference including tactile information. That is, the multipoint conference system of the reference embodiment uses one or more predetermined channels of the multiple channels of a specified audio transmission method as channels for audio signals, and uses one or more predetermined channels different from the channels for audio signals of the multiple channels of the specified audio transmission method as channels for vibration signals. The multipoint conference system of the reference embodiment and each device included in the system will be described below.
[0011] A multipoint conference system 10 of the reference embodiment includes a multipoint conference control device 100 and terminal devices 200-m, as shown in FIG. 1. In the multipoint conference system 10, M is an integer of 2 or more, and m is an integer of 1 to M. The multipoint conference control device 100 and each terminal device 200-m are connected via each transmission path 300-m of a communication network. The communication network is, for example, the Internet, and is capable of transmitting signals between a plurality of devices in almost real time. In the following, the point where the terminal device 200-m is located is referred to as point m. Note that a reference number with a "-" is a representation in which a number is connected with a "-" and does not represent a number subtracted from a number.
[0012] [Terminal device 200-m] The terminal device 200-m is a terminal device located at a point m, and is composed of, for example, a PC and peripheral devices connected to the PC, and includes a transmitting device 210-m and a receiving device 220-m as shown in Fig. 2. Here, the transmitting device 210-m will be explained first. The receiving device 220-m will be explained after the multipoint conference control device 100 has been explained.
[0013] [Transmitting device 210-m] The transmitting device 210-m includes a video acquisition unit 211-m, an audio acquisition unit 212-m, a vibration acquisition unit 213-m, and a transmission signal generation unit 214-m, as illustrated in Fig. 2. The transmitting device 210-m performs each process of steps S211, S212, S213, and S214 illustrated in Fig. 3 for each predetermined time interval. The predetermined time interval is a time interval for generating a terminal transmission signal SA-m determined by the encoding processing unit or the packetization processing unit, and is, for example, an interval of several tens of ms in length.
[0014] [Video acquisition unit 211-m] The image acquisition unit 211-m is, for example, a camera built into a PC or a camera connected to a PC. The image acquisition unit 211-m captures an image of a point m, and outputs a video signal corresponding to the captured image to the transmission signal generation unit 214-m (step S211).
[0015] [Acoustic acquisition section 212-m] The sound acquisition unit 212-m is, for example, a microphone built into the PC or a microphone connected to the PC. The sound acquisition unit 212-m collects a sound emitted at the point m and outputs an acoustic signal corresponding to the collected sound to the transmission signal generation unit 214-m (step S212).
[0016] [Vibration acquisition section 213-m] The vibration acquisition unit 213-m is, for example, a vibration measurement sensor connected to a PC. The vibration acquisition unit 213-m acquires the vibration generated at the point m, and outputs a vibration signal corresponding to the acquired vibration to the transmission signal generation unit 214-m (step S213).
[0017] [Output signal generation unit 214-m] The transmission signal generating unit 214-m receives the video signal output by the video acquiring unit 211-m, the audio signal output by the audio acquiring unit 212-m, and the vibration signal output by the vibration acquiring unit 213-m. The transmission signal generating unit 214-m generates a video bit string that is a bit string of a predetermined video transmission method including the input video signal, generates an audio vibration bit string that is a bit string of a predetermined audio transmission method corresponding to multiple channels including the input audio signal and the input vibration signal, obtains a terminal transmission signal SA-m that includes the generated video bit string and the generated audio vibration bit string, and outputs it to the multipoint conference control device 100 (step S214).
[0018] The predetermined video transmission method is, for example, a standard for transmitting one-channel video signals, and may be lossy encoding such as MPEG-2 Visual or MPEG-4 Visual, lossless encoding, or non-compression. The transmission signal generator 214-m generates a video bitstream by, for example, encoding the input video signal in accordance with the predetermined video transmission method or storing the video signal in a bitstream.
[0019] The predetermined acoustic transmission method corresponding to multiple channels is, for example, a standard for transmitting a 2-channel stereo acoustic signal or a 5.1-channel surround acoustic signal, and may be lossy encoding such as MPEG-4 AAC, lossless encoding such as MPEG-4 ALS, or non-compression. The transmission signal generating unit 214-m converts the input acoustic signal into a signal of a predetermined channel (hereinafter also referred to as a "channel for acoustic signal") among the multiple channels of the predetermined acoustic transmission method, converts the input vibration signal into a signal of a predetermined channel other than the channel for acoustic signal among the multiple channels of the predetermined acoustic transmission method (hereinafter also referred to as a "channel for vibration signal"), and generates an acoustic vibration bit string by performing encoding processing corresponding to the predetermined acoustic transmission method or storing processing into a bit string on the multiple channel signals. Note that although the acoustic vibration bit string is actually a bit string of a predetermined acoustic transmission method as described above, it is a bit string including both an acoustic signal and a vibration signal, and is therefore conveniently referred to as an "acoustic vibration bit string" in this specification. Examples of using a 2-channel stereo acoustic transmission method and an example of using a 5.1-channel surround acoustic transmission method will be described later, but in short, the transmission signal generation unit 214-m sets one or more predetermined channels out of the multiple channels of a specified acoustic transmission method as channels for acoustic signals, and sets one or more predetermined channels out of the multiple channels of the specified acoustic transmission method different from the channels for acoustic signals as channels for vibration signals, stores acoustic signals in the channels for acoustic signals, and outputs a terminal transmission signal SA-m in which the vibration signals are stored in the channels for vibration signals to the multipoint conference control device 100.
[0020] [Example of using a 2-channel stereo audio transmission method] When a two-channel stereo acoustic transmission method is used, for example, a left channel (L channel) that is a predetermined one of the two channels is used as a channel for an acoustic signal, and a right channel (R channel) that is a predetermined one of the two channels different from the channel for an acoustic signal is used as a channel for a vibration signal. That is, when a two-channel stereo acoustic transmission method is used, the transmission signal generating unit 214-m may set the acoustic signal to a left channel signal and the vibration signal to a right channel signal, as exemplified in FIG. 4.
[0021] When using a two-channel stereo sound transmission method, the channel for the sound signal itself is one channel, but the sound acquisition unit 212-m may be one microphone or two or more microphones. That is, the number of microphones included in the sound acquisition unit 212-m and the number of channels for the sound signal may be the same or different. When the sound acquisition unit 212-m is one microphone, the transmission signal generation unit 214-m may directly use one input sound signal as a left channel signal. When the sound acquisition unit 212-m is two or more microphones, for example, the transmission signal generation unit 214-m may use a signal obtained by mixing two or more input sound signals as a left channel signal, or may use a signal selected from two or more input sound signals as a left channel signal.
[0022] Similarly, when a two-channel stereo acoustic transmission method is used, the channel for the vibration signal itself is one channel, but the vibration acquisition unit 213-m may be one vibration measurement sensor or two or more vibration measurement sensors. That is, the number of vibration measurement sensors included in the vibration acquisition unit 213-m and the number of channels for the vibration signal may be the same or different. When the vibration acquisition unit 213-m is one vibration measurement sensor, the transmission signal generation unit 214-m may directly use one input vibration signal as a right channel signal. When the vibration acquisition unit 213-m is two or more vibration measurement sensors, for example, the transmission signal generation unit 214-m may use a signal obtained by mixing two or more input vibration signals as a right channel signal, or may use a signal selected from two or more input vibration signals as a right channel signal.
[0023] [Example of using 5.1 channel surround sound transmission method] When using a 5.1 channel surround sound transmission method, for example, two predetermined channels out of six channels, the left front channel and the right front channel, are used as channels for sound signals, and four predetermined channels out of six channels different from the channels for sound signals, the front channel, the left rear channel, the right rear channel, and the deep bass channel, are used as channels for vibration signals. That is, when using a 5.1 channel surround sound transmission method, as illustrated in Fig. 5, the transmission signal generating unit 214-m sets the two sound signals of the left channel and the right channel as left front channel and right front channel signals, respectively, and sets the first to fourth four vibration signals as front channel signals, left rear channel signals, right rear channel signals, and deep bass channel signals, respectively. When using a 5.1 channel surround sound transmission method, the number of microphones included in the sound acquisition unit 212-m and the number of channels for sound signals may be the same or different, and the number of vibration measurement sensors included in the vibration acquisition unit 213-m and the number of channels for vibration signals may be the same or different, as in the case of using a 2 channel stereo sound transmission method. Note that the transmission signal generation unit 214-m may not store a signal in any of the multiple channels for sound signals, and may not store a signal in any of the multiple channels for vibration signals.
[0024] [Whether video is transmitted or not] It is not essential that the sending signal generating unit 214-m includes a video bitstream in the terminal sending signal SA-m. For example, the transmitting side device 210-m may not include the video acquiring unit 211-m, and the transmitting side device 210-m may not perform step S211, and the sending signal generating unit 214-m may output the terminal sending signal SA-m that does not include a video bitstream to the multipoint conference control device 100. Also, for example, as in the multipoint conference system of Non-Patent Document 1, the user may be allowed to select whether or not the transmitting side device 210-m sends out video. In other words, the transmitting side device 210-m may be provided with a transmission signal selection unit 215-m including a switch (hereinafter referred to as a "video transmission ON / OFF switch") that accepts user input regarding whether or not the transmitting side device 210-m will transmit video, and the transmission signal generation unit 214-m may operate to determine whether or not to include a video bit string in the terminal transmission signal SA-m in accordance with the input accepted by the video transmission ON / OFF switch of the transmission signal selection unit 215-m.
[0025] The switch may be a hardware switch provided in the terminal device 200-m, but may also be a software switch provided on a screen displayed on a display device built into or connected to the terminal device 200-m, and may be operated by a user using a mouse provided in the terminal device 200-m or by touching a display device that is a touch screen. This also applies to each of the following switches.
[0026] [Sound transmission or not] It is not essential that the transmission signal generating unit 214-m includes an acoustic signal in the acoustic vibration bit string. For example, the transmitting side device 210-m may not include the acoustic acquiring unit 212-m, and the transmitting side device 210-m may not perform step S212. The transmission signal generating unit 214-m may generate an acoustic vibration bit string that includes the input vibration signal but does not include an acoustic signal, and output a terminal transmission signal SA-m that includes the generated acoustic vibration bit string to the multipoint conference control device 100. Also, for example, as in the multipoint conference system of Non-Patent Document 1, the user may be allowed to select whether or not the transmitting side device 210-m transmits sound. That is, the transmitting side device 210-m may be provided with a transmission signal selection unit 215-m including a switch (hereinafter referred to as an "audio transmission ON / OFF switch") that accepts a user's input regarding whether the transmitting side device 210-m transmits sound, and the transmission signal generation unit 214-m may operate to determine whether or not to include an acoustic signal in an acoustic vibration bitstream according to the input accepted by the acoustic transmission ON / OFF switch of the transmission signal selection unit 215-m. Note that, when not transmitting sound, the transmission signal generation unit 214-m may include an all-zero signal in a predetermined channel for an acoustic signal among multiple channels of a predetermined acoustic transmission method, and then perform encoding processing or storage processing into a bitstream corresponding to the predetermined acoustic transmission method on the signals of the multiple channels to generate an acoustic vibration bitstream, and include the generated acoustic vibration bitstream in the terminal transmission signal SA-m.
[0027] [Whether vibration is sent or not] It is not essential that the sending signal generating unit 214-m includes a vibration signal in the acoustic vibration bit string. For example, the sending side device 210-m may not include the vibration acquiring unit 213-m, and the sending side device 210-m may not perform step S213. The sending signal generating unit 214-m may generate an acoustic vibration bit string that includes the input acoustic signal but does not include a vibration signal, and output a terminal sending signal SA-m that includes the generated acoustic vibration bit string to the multipoint conference control device 100. Also, for example, the user may be allowed to select whether the sending side device 210-m sends out vibration. That is, the transmitting side device 210-m may be provided with a transmission signal selection unit 215-m including a switch (hereinafter referred to as a "vibration transmission ON / OFF switch") that accepts a user's input regarding whether the transmitting side device 210-m transmits vibration, and the transmission signal generation unit 214-m may operate to determine whether or not to include a vibration signal in an acoustic vibration bit string according to the input accepted by the vibration transmission ON / OFF switch of the transmission signal selection unit 215-m. Note that, when not transmitting vibration, the transmission signal generation unit 214-m may include an all-zero signal in a predetermined channel for a vibration signal among multiple channels of a predetermined acoustic transmission method, and then perform encoding processing or storage processing into a bit string corresponding to the predetermined acoustic transmission method on the signals of the multiple channels to generate an acoustic vibration bit string, and include the generated acoustic vibration bit string in the terminal transmission signal SA-m.
[0028] [When neither sound nor vibration is sent] When the transmission signal generating unit 214-m does not transmit sound or vibration, it may include all-zero signals in both a channel for a predetermined acoustic signal among the multiple channels of a predetermined acoustic transmission method and a channel for a predetermined vibration signal among the multiple channels of a predetermined acoustic transmission method, and then generate an acoustic vibration bit string by performing encoding processing or storage processing into a bit string corresponding to the predetermined acoustic transmission method on the signals of the multiple channels, and include the generated acoustic vibration bit string in the terminal transmission signal SA-m, or it may not include the acoustic vibration bit string in the terminal transmission signal SA-m to be output.
[0029] [Multipoint conference control device 100] The multipoint conference control device 100 includes a conference signal generating unit 110, as shown in Fig. 6. Terminal transmission signals SA-1 to SA-M output from the terminal devices 200-1 to 200-M connected via the transmission paths 300-1 to 300-M are input to the multipoint conference control device 100. The multipoint conference control device 100 obtains and outputs a terminal-directed signal SB by performing the process of step S110 shown in Fig. 7 for each predetermined time period.
[0030] [Conference signal generator 110] Terminal transmission signals SA-1 to SA-M are input to the conference signal generating unit 110. The conference signal generating unit 110 performs, for example, each process from step S110-1 to step S110-6 shown in Fig. 8 to obtain and output a terminal-directed signal SB from the input terminal transmission signals SA-1 to SA-M (step S110).
[0031] The conference signal generating unit 110 generates a video signal for output including a video signal included in the video bit string of each point included in each of the terminal sending signals SA-1 to SA-M (step S110-1). For example, similar to the multipoint conference system of Non-Patent Document 1, the conference signal generating unit 110 may obtain a reduced video by reducing the size of the video of each point, generate one video including the reduced video of each point, and generate a video signal corresponding to the generated video. When there is a terminal sending signal that does not include a video bit string among the terminal sending signals SA-1 to SA-M, the conference signal generating unit 110 may generate a video signal for output including a video signal of each point only from the terminal sending signals including a video bit string. After performing step S110-1, the conference signal generating unit 110 generates a video bit string that is a bit string of a predetermined video transmission method including the video signal obtained in step S110-1 (step S110-2).
[0032] The conference signal generating unit 110 generates an output acoustic signal including an acoustic signal included in the acoustic vibration bit string of each point included in each of the terminal transmission signals SA-1 to SA-M (step S110-3). The conference signal generating unit 110 may generate one acoustic signal by mixing the acoustic signals of each point, as in the multipoint conference system of Non-Patent Document 1, for example. When there is a terminal transmission signal in which an acoustic signal is not included in the acoustic vibration bit string of the terminal transmission signals SA-1 to SA-M, the conference signal generating unit 110 may generate an output acoustic signal including an acoustic signal of each point only for the terminal transmission signal in which an acoustic signal is included in the acoustic vibration bit string. When there are multiple channels for acoustic signals, the conference signal generating unit 110 may generate an output acoustic signal including an acoustic signal of each point for each channel.
[0033] Furthermore, the conference signal generating unit 110 generates an output vibration signal including a vibration signal included in the acoustic vibration bit string of each point included in each of the terminal sending signals SA-1 to SA-M (step S110-4). The conference signal generating unit 110 may generate one vibration signal by, for example, mixing the vibration signals of each point. When there is a terminal sending signal in which the acoustic vibration bit string of the terminal sending signals SA-1 to SA-M does not include a vibration signal, the conference signal generating unit 110 may generate an output vibration signal including a vibration signal of each point only for the terminal sending signal in which the acoustic vibration bit string includes a vibration signal. When there are multiple channels for vibration signals, the conference signal generating unit 110 may generate an output vibration signal including a vibration signal of each point for each channel.
[0034] After performing steps S110-3 and S110-4, the conference signal generating unit 110 stores the acoustic signal obtained in step S110-3 in a predetermined channel for acoustic signals among the multiple channels of the predetermined acoustic transmission method, and stores the vibration signal obtained in step S110-4 in a predetermined channel for vibration signals different from the channel for acoustic signals among the multiple channels of the predetermined acoustic transmission method, thereby generating an acoustic vibration bit string that is a bit string of the predetermined acoustic transmission method corresponding to the multiple channels (step S110-5). After performing steps S110-2 and S110-5, the conference signal generating unit 110 obtains a terminal-oriented signal SB including the video bit string generated in step S110-2 and the acoustic vibration bit string generated in step S110-5, and outputs it to each terminal device, that is, outputs it to each of the terminal devices 200-1 to 200-M via each transmission path 300-1 to 300-M (step S110-6).
[0035] If no video signal is included in any of the terminal transmission signals, the conference signal generating unit 110 performs step S110-5 without performing steps S110-1 and S110-2, and then performs step S110-5, and in step S110-6, obtains a terminal-directed signal SB including the acoustic vibration bit string generated in step S110-5 and outputs it to each terminal device. In short, the conference signal generating unit 110 stores an acoustic signal based on an acoustic signal input from each point in a channel for an acoustic signal that is one or more predetermined channels among the multiple channels of a predetermined acoustic transmission method, and outputs a terminal-directed signal SB to each terminal device in which a vibration signal based on a vibration signal input from each point is stored in a channel for a vibration signal that is one or more predetermined channels different from the channel for an acoustic signal among the multiple channels of the predetermined acoustic transmission method.
[0036] [Receiving device 220-m] The receiving side device 220-m includes a presentation signal generating unit 224-m, a video presentation unit 221-m, an audio presentation unit 222-m, and a vibration presentation unit 223-m, as illustrated in Fig. 2. The receiving side device 220-m receives a terminal-directed signal SB output by the multipoint conference control device 100 connected via each transmission path 300-m. The receiving side device 220-m performs each process of steps S224, S221, S222, and S223 illustrated in Fig. 9 for each predetermined time period.
[0037] [Presentation signal generation unit 224-m] The presentation signal generating unit 224-m receives the terminal-directed signal SB as input. The presentation signal generating unit 224-m obtains a video signal from a video bit string, which is a bit string of a predetermined video transmission method included in the terminal-directed signal SB, and outputs the video signal to the video presentation unit 221-m, obtains a signal of a channel for an audio signal from an audio vibration bit string, which is a bit string of a predetermined audio transmission method included in the terminal-directed signal SB, and outputs the signal as an audio signal to the audio presentation unit 222-m, and obtains a signal of a channel for a vibration signal from an audio vibration bit string, which is a bit string of a predetermined audio transmission method included in the terminal-directed signal SB, and outputs the signal as a vibration signal to the vibration presentation unit 223-m (step S224).
[0038] [Video presentation section 221-m] The video presentation unit 221-m is, for example, a display device built into a PC or a display device connected to a PC. The video presentation unit 221-m receives the video signal output by the presentation signal generation unit 224-m. The video presentation unit 221-m displays a video corresponding to the input video signal, that is, presents the video (step S221).
[0039] [Sound presentation section 222-m] The audio presentation unit 222-m is, for example, a speaker built into the PC or a speaker connected to the PC. The audio presentation unit 222-m receives an audio signal output by the presentation signal generation unit 224-m. The audio presentation unit 222-m generates a sound corresponding to the input audio signal, that is, presents the sound (step S222).
[0040] [Vibration presentation section 223-m] The vibration presentation unit 223-m is, for example, a vibrator connected to a PC. The vibration presentation unit 223-m receives a vibration signal output by the presentation signal generation unit 224-m. The vibration presentation unit 223-m generates a vibration corresponding to the input vibration signal, that is, presents a vibration (step S223).
[0041] [Whether or not video is shown] It is not essential that the receiving device 220-m presents a video. For example, the receiving device 220-m may not include the video presenting unit 221-m, the presentation signal generating unit 224-m may not obtain a video signal from the terminal-directed signal SB, and the receiving device 220-m may not perform step S221. Also, for example, the receiving device 220-m may be configured to allow a user to select whether or not the receiving device 220-m presents a video. That is, the receiving device 220-m may be configured to include a presentation signal selecting unit 225-m including a switch (hereinafter referred to as a "video presentation ON / OFF switch") that accepts an input by a user regarding whether or not the receiving device 220-m presents a video, and the presentation signal generating unit 224-m may operate to output or not output a video signal according to the input accepted by the video presentation ON / OFF switch of the presentation signal selecting unit 225-m.
[0042] [Whether or not sound is presented] It is not essential that the receiving device 220-m presents a sound. For example, the receiving device 220-m may not include the audio presentation unit 222-m, the presentation signal generation unit 224-m may not obtain an audio signal from the terminal-directed signal SB, and the receiving device 220-m may not perform step S222. Also, for example, the receiving device 220-m may be configured to allow a user to select whether or not the receiving device 220-m presents a sound. That is, the receiving device 220-m may be configured to include a presentation signal selection unit 225-m including a switch (hereinafter referred to as an "audio presentation ON / OFF switch") that accepts an input by a user regarding whether or not the receiving device 220-m presents a sound, and the presentation signal generation unit 224-m may operate to output or not output an audio signal according to the input accepted by the audio presentation ON / OFF switch of the presentation signal selection unit 225-m.
[0043] [Whether or not vibration is displayed] It is not essential that the receiving device 220-m presents vibration. For example, the receiving device 220-m may not include the vibration presentation unit 223-m, the presentation signal generation unit 224-m may not obtain a vibration signal from the terminal-directed signal SB, and the receiving device 220-m may not perform step S223. Also, for example, the receiving device 220-m may be configured so that the user can select whether or not to present vibration. The receiving device 220-m may be configured to include a presentation signal selection unit 225-m including a switch (hereinafter, referred to as a "vibration presentation ON / OFF switch") that accepts an input by the user regarding whether or not the receiving device 220-m presents vibration, and the presentation signal generation unit 224-m may operate to output or not output a vibration signal according to the input accepted by the vibration presentation ON / OFF switch of the presentation signal selection unit 225-m.
[0044] First Embodiment The multipoint conference system 10 of the reference embodiment substitutes a channel originally prepared for an acoustic signal in a predetermined acoustic transmission method as a channel for a vibration signal. Therefore, if a terminal device that reproduces a multi-channel acoustic signal according to a predetermined acoustic transmission method is used as a terminal device of the multipoint conference system 10 of the reference embodiment instead of the above-mentioned terminal device 200-m, the terminal device will output a vibration signal of a channel for a vibration signal to a speaker for an acoustic signal, and the speaker may emit an unexpected sound, break down, or be destroyed. In order to eliminate these possibilities, the multipoint conference control device outputs a terminal-directed signal suitable for each terminal device in the multipoint conference system of the first embodiment. Below, the multipoint conference system of the first embodiment and each device included in the system will be described, focusing on the differences from the reference embodiment.
[0045] As illustrated in FIG. 10, the multipoint conference system 11 of the first embodiment includes a multipoint conference control device 101, a tactile-enabled terminal device 200-m, and a non-tactile-enabled terminal device 400-n. In the first embodiment, M is an integer of 1 or more, m is an integer of 1 to M, N is an integer of 1 or more, and n is an integer of M+1 to M+N. The multipoint conference control device 101 and each tactile-enabled terminal device 200-m are connected via each transmission path 300-m of the communication network. The multipoint conference control device 101 and each non-tactile-enabled terminal device 400-n are connected via each transmission path 500-n of the communication network. The communication network is, for example, the Internet, and can transmit signals between multiple devices in almost real time. In the following, the point where the tactile-enabled terminal device 200-m is located is called point m, and the point where the non-tactile-enabled terminal device 400-n is located is called point n.
[0046] [Tactile terminal device 200-m] The haptic-enabled terminal device 200-m is similar to the terminal device 200-m of the reference embodiment, and the configuration and operation of each part of the haptic-enabled terminal device 200-m are similar to those described in the description of the terminal device 200-m of the reference embodiment. The name "haptic-enabled terminal device" is adopted to clearly distinguish it from the "non-haptic-enabled terminal device" included in the multipoint conference system 11 of the first embodiment. For the same reason, the transmitting side device 210-m of the reference embodiment is called the haptic-enabled transmitting side device 210-m in the first embodiment, the receiving side device 220-m of the reference embodiment is called the haptic-enabled receiving side device 220-m in the first embodiment, the terminal sending signal SA-m in the reference embodiment is called the haptic-enabled terminal sending signal SA-m in the first embodiment, and the terminal-directed signal SB in the reference embodiment is called the haptic-enabled terminal-directed signal SB in the first embodiment. That is, the haptic-enabled transmitting device 210-m outputs a haptic-enabled terminal sending signal SA-m to the multipoint conference control device 101 via the transmission path 300-m. Also, a haptic-enabled terminal-directed signal SB is input to the haptic-enabled receiving device 220-m from the multipoint conference control device 101 via the transmission path 300-m.
[0047] [Non-tactile terminal device 400-n] The non-tactile-enabled terminal device 400-n is a terminal device located at point n, and is composed of, for example, a PC and a peripheral device connected to the PC, or a PC alone, and includes a non-tactile-enabled transmitting device 410-n and a non-tactile-enabled receiving device 420-n as shown in Fig. 11. First, the non-tactile-enabled transmitting device 410-n will be described. The non-tactile-enabled receiving device 420-n will be described after the multipoint conference control device 101 has been described.
[0048] [Non-haptic transmitting device 410-n] The haptic incompatible transmitting device 410-n includes a video acquiring unit 411-n, a sound acquiring unit 412-n, and a transmission signal generating unit 414-n, as illustrated in Fig. 11. The haptic incompatible transmitting device 410-n performs the processes of steps S411, S412, and S414 illustrated in Fig. 12 for each predetermined time interval.
[0049] [Video acquisition unit 411-n] The image acquisition unit 411-n is, for example, a camera built into a PC or a camera connected to a PC. The image acquisition unit 411-n captures an image of a point n, and outputs a video signal corresponding to the captured image to the transmission signal generation unit 414-n (step S411).
[0050] [Acoustic acquisition unit 412-n] The sound acquisition unit 412-n is, for example, a microphone built into the PC or a microphone connected to the PC. The sound acquisition unit 412-n collects a sound emitted at the point n and outputs an acoustic signal corresponding to the collected sound to the transmission signal generation unit 414-n (step S412).
[0051] [Output signal generation unit 414-n] The transmission signal generation unit 414-n receives the video signal output by the video acquisition unit 411-n and the audio signal output by the audio acquisition unit 412-n. The transmission signal generation unit 414-n generates a video bit string that is a bit string of a predetermined video transmission method including the input video signal, generates an audio bit string that is a bit string of a predetermined audio transmission method corresponding to multiple channels including the input audio signal, obtains a tactile non-compatible terminal transmission signal SC-n that includes the generated video bit string and the generated audio bit string, and outputs it to the multipoint conference control device 101 (step S414).
[0052] The transmission signal generation unit 414-n, like the transmission signal generation unit 214-m of the haptic-compatible transmission device 210-m, generates a video bitstream by performing encoding processing on the input video signal corresponding to a predetermined video transmission method or storing processing into a bitstream.
[0053] The transmission signal generation unit 414-n generates an audio bit string by treating the input audio signal as a signal for the channel for the above-mentioned audio signal among the multiple channels of a predetermined audio transmission method, and performing encoding processing or storing processing into a bit string corresponding to the predetermined audio transmission method on the signals of the multiple channels without including a substantial signal in the channel for the above-mentioned vibration signal among the multiple channels of the predetermined audio transmission method. For example, in the case of generating an audio bit string that always includes information of multiple channels set with a predetermined audio transmission method, the transmission signal generation unit 414-n generates an audio bit string by treating the input audio signal as a signal for the channel for the above-mentioned audio signal among the multiple channels of the predetermined audio transmission method, and performing encoding processing or storing processing into a bit string corresponding to the predetermined audio transmission method on the signals of the multiple channels as an all-zero signal as a signal for the channel for the above-mentioned vibration signal among the multiple channels of the predetermined audio transmission method. For example, in the case where a specified acoustic transmission method is to generate an acoustic bitstream including information of only the necessary channels, the transmission signal generation unit 414-n treats the input acoustic signal as a signal of the channel for acoustic signals described above among the multiple channels of the specified acoustic transmission method, performs encoding processing corresponding to the specified acoustic transmission method or storage processing into a bitstream on the signal of the channel for acoustic signals, and generates an acoustic bitstream by including information indicating that the channel for acoustic signals is the only necessary channel.
[0054] [Whether video is transmitted or not] It is not essential for the transmission signal generation unit 414-n to include a video bitstream in the non-haptic-compatible terminal transmission signal SC-n. For example, the non-haptic-compatible transmitting side device 410-n may not include the video acquisition unit 411-n, and the non-haptic-compatible transmitting side device 410-n may not perform step S411, and the transmission signal generation unit 414-n may output the non-haptic-compatible terminal transmission signal SC-n that does not include a video bitstream to the multipoint conference control device 101. Also, for example, as in the multipoint conference system of Non-Patent Document 1, the user may be allowed to select whether or not the non-haptic-compatible transmitting side device 410-n transmits video. In other words, the non-haptic-compatible transmitting side device 410-n may be provided with a transmission signal selection unit 415-n including a video transmission ON / OFF switch which is a switch that accepts input from a user as to whether or not the non-haptic-compatible transmitting side device 410-n will transmit video, and the transmission signal generation unit 414-n may operate to determine whether or not to include a video bit string in the non-haptic-compatible terminal transmission signal SC-n according to the input accepted by the video transmission ON / OFF switch of the transmission signal selection unit 415-n.
[0055] [Sound transmission or not] It is not essential for the transmission signal generation unit 414-n to include an audio bit string in the non-haptic-capable terminal transmission signal SC-n. For example, the non-haptic-capable transmitting side device 410-n may not include the audio acquisition unit 412-n, and the non-haptic-capable transmitting side device 410-n may not perform step S412, and the transmission signal generation unit 414-n may output the non-haptic-capable terminal transmission signal SC-n that does not include an audio bit string to the multipoint conference control device 101. Also, for example, as in the multipoint conference system of Non-Patent Document 1, the user may be allowed to select whether or not the non-haptic-capable transmitting side device 410-n transmits sound. That is, the non-haptic-enabled transmitting side device 410-n may be provided with a transmission signal selection unit 415-n including an audio transmission ON / OFF switch that receives a user's input as to whether or not the non-haptic-enabled transmitting side device 410-n transmits sound, and the transmission signal generation unit 414-n may operate to determine whether or not to include an audio bit string in the non-haptic-enabled terminal transmission signal SC-n according to the input received by the audio transmission ON / OFF switch of the transmission signal selection unit 415-n. Note that, when sound is not to be transmitted, the transmission signal generation unit 414-n may include an all-zero signal in the channel for the audio signal, and then perform encoding processing or bit string storage processing corresponding to a predetermined audio transmission method on the signals of the multiple channels to generate an audio bit string, and include the generated audio bit string in the non-haptic-enabled terminal transmission signal SC-n.
[0056] [Multipoint conference control device 101] The multipoint conference control device 101 includes a conference signal generating unit 111, as shown in Fig. 13. The multipoint conference control device 101 receives tactile-enabled terminal sending signals SA-1 to SA-M output from the tactile-enabled terminal devices 200-1 to 200-M connected via the transmission paths 300-1 to 300-M, and non-tactile-enabled terminal sending signals SC-(M+1) to SC-(M+N) output from the non-tactile-enabled terminal devices 400-(M+1) to 400-(M+N) connected via the transmission paths 500-(M+1) to 500-(M+N). The multipoint conference control device 101 performs the process of step S111 shown in Fig. 14 for each predetermined time period.
[0057] [Conference signal generator 111] Haptic-enabled terminal transmission signals SA-1 to SA-M and non-haptic-enabled terminal transmission signals SC-(M+1) to SC-(M+N) are input to the conference signal generation unit 111. The conference signal generation unit 111 performs the processes of steps S111-1 to S111-8 shown in Fig. 15, for example, to obtain and output a haptic-enabled terminal-directed signal SB and a non-haptic terminal-directed signal SD from the input haptic-enabled terminal transmission signals SA-1 to SA-M and non-haptic terminal transmission signals SC-(M+1) to SC-(M+N) (step S111).
[0058] The conference signal generating unit 111 generates a video signal for output including a video signal included in the video bit strings of each point included in the haptic-enabled terminal sending signals SA-1 to SA-M and the haptic-incompatible terminal sending signals SC-(M+1) to SC-(M+N) (step S111-1). The conference signal generating unit 111 may, for example, obtain reduced images by reducing the size of the images of each point, generate one image including the reduced images of each point, and generate a video signal corresponding to the generated images, similar to the multipoint conference system of Non-Patent Document 1. When there is a haptic-enabled terminal transmission signal that does not include a video bitstream among the haptic-enabled terminal transmission signals SA-1 to SA-M, or when there is a non-haptic-enabled terminal transmission signal that does not include a video bitstream among the non-haptic-enabled terminal transmission signals SC-(M+1) to SC-(M+N), the conference signal generation unit 111 only needs to generate a video signal for output that includes a video signal at each point from the haptic-enabled terminal transmission signals that include a video bitstream and the non-haptic-enabled terminal transmission signals. After performing step S111-1, the conference signal generation unit 111 generates a video bitstream that is a bitstream of a predetermined video transmission method that includes the video signal obtained in step S111-1 (step S111-2).
[0059] The conference signal generator 111 also generates an output acoustic signal including an acoustic signal included in the acoustic vibration bit strings of each point included in each of the haptic-enabled terminal transmission signals SA-1 to SA-M and an acoustic signal included in the acoustic bit strings of each point included in each of the haptic-incompatible terminal transmission signals SC-(M+1) to SC-(M+N) (step S111-3). The conference signal generator 111 may generate one acoustic signal by mixing the acoustic signals of each point, as in the multipoint conference system of Non-Patent Document 1, for example. When there is a haptic-enabled terminal sending signal that does not include an acoustic signal in the acoustic vibration bit string of the haptic-enabled terminal sending signals SA-1 to SA-M, or when there is a haptic-enabled terminal sending signal that does not include an acoustic bit string (i.e., a haptic-enabled terminal sending signal that does not include an acoustic signal) in the haptic-unenabled terminal sending signals SC-(M+1) to SC-(M+N), the conference signal generating unit 111 may generate an output acoustic signal that includes an acoustic signal of each point only for haptic-enabled terminal sending signals that include an acoustic signal in the acoustic vibration bit string and haptic-enabled non-terminal sending signals that include an acoustic bit string (i.e., only haptic-enabled terminal sending signals that include an acoustic signal and haptic-enabled non-terminal sending signals that include an acoustic signal).When there are multiple channels for acoustic signals, the conference signal generating unit 111 may generate an output acoustic signal that includes an acoustic signal of each point for each channel.
[0060] Furthermore, the conference signal generating unit 111 generates an output vibration signal including a vibration signal included in the acoustic vibration bit string of each point included in each of the haptic-compatible terminal sending signals SA-1 to SA-M (step S111-4). The conference signal generating unit 111 may generate one vibration signal by, for example, mixing the vibration signals of each point. When there is a haptic-compatible terminal sending signal in which a vibration signal is not included in the acoustic vibration bit string of the haptic-compatible terminal sending signals SA-1 to SA-M, the conference signal generating unit 111 may generate an output vibration signal including a vibration signal of each point only for the haptic-compatible terminal sending signal in which a vibration signal is included in the acoustic vibration bit string. When there are multiple channels for vibration signals, the conference signal generating unit 111 may generate an output vibration signal including a vibration signal of each point for each channel.
[0061] After performing steps S111-3 and S111-4, the conference signal generating unit 111 stores the acoustic signal obtained in step S111-3 in a predetermined channel for acoustic signals among the multiple channels of the predetermined acoustic transmission method, and stores the vibration signal obtained in step S111-4 in a predetermined channel for vibration signals different from the channel for acoustic signals among the multiple channels of the predetermined acoustic transmission method, thereby generating an acoustic vibration bit string that is a bit string of the predetermined acoustic transmission method corresponding to the multiple channels (step S111-5). After performing steps S111-2 and S111-5, the conference signal generating unit 111 obtains a haptic-compatible terminal-oriented signal SB including the video bit string generated in step S111-2 and the acoustic vibration bit string generated in step S111-5, and outputs it to each haptic-compatible terminal device, that is, outputs it to each of the haptic-compatible terminal devices 200-1 to 200-M via each transmission path 300-1 to 300-M (step S111-6).
[0062] Steps S111-1 to S111-6 performed by the conference signal generating unit 111 are similar to steps S110-1 to S110-6 performed by the conference signal generating unit 110 of the reference embodiment, but the conference signal generating unit 111 also performs the following steps S111-7 and S111-8. After performing step S111-3, the conference signal generating unit 111 stores the acoustic signal obtained in step S111-3 in a predetermined channel for acoustic signals among the multiple channels of a predetermined acoustic transmission method, and stores an all-zero signal or the acoustic signal obtained in step S111-3 in a predetermined channel for vibration signals different from the channel for acoustic signals among the multiple channels of the predetermined acoustic transmission method, thereby generating an acoustic bit string that is a bit string of a predetermined acoustic transmission method corresponding to the multiple channels (step S111-7). After performing steps S111-2 and S111-7, the conference signal generating unit 111 obtains a signal SD for a non-haptic terminal including the video bit string generated in step S111-2 and the audio bit string generated in step S111-7, and outputs it to each non-haptic terminal device, that is, outputs it to each of the non-haptic terminal devices 400-(M+1) to 400-(M+N) via each transmission path 500-(M+1) to 500-(M+N) (step S111-8). Note that the all-zero signal is a signal whose amplitude is 0 at all times, and is an example of a signal that does not substantially generate sound or vibration even when input to the audio presentation unit or vibration presentation unit. In other words, the conference signal generating unit 110 may use a signal that is equivalent to substantially no signal, such as a signal with a small amplitude, instead of an all-zero signal, as long as the signal does not substantially generate sound or vibration even when input to the audio presentation unit or vibration presentation unit. The same applies to the subsequent all-zero signals.
[0063] If none of the haptic-enabled terminal output signals and the non-haptic-enabled terminal signals contain a video signal, the conference signal generating unit 111 does not perform steps S111-1 and S111-2, but instead performs step S111-5, and then, in step S111-6, obtains a signal SB for the haptic-enabled terminal including the acoustic vibration bit string generated in step S111-5 and outputs it to each haptic-enabled terminal device, and after performing step S111-7, obtains a signal SD for the non-haptic terminal including the acoustic bit string generated in step S111-7 and outputs it to each non-haptic terminal device, in step S111-8. In short, the conference signal generating unit 111 outputs to each tactile-compatible terminal device a signal SB for a tactile-compatible terminal in which an acoustic signal based on the acoustic signal input from each point is stored in a channel for acoustic signals, which is one or more predetermined channels among the multiple channels of a predetermined acoustic transmission method, and a vibration signal based on the vibration signal input from each point is stored in a channel for vibration signals, which is one or more predetermined channels different from the channel for acoustic signals among the multiple channels of a predetermined acoustic transmission method, and outputs to each non-tactile-compatible terminal device a signal SD for a non-tactile-compatible terminal in which an acoustic signal based on the acoustic signal input from each point is stored in the channel for acoustic signals, and an all-zero signal or an acoustic signal based on the acoustic signal input from each point is stored in the channel for vibration signals.
[0064] [Non-haptic receiving device 420-n] The non-haptic-sense-capable receiving device 420-n includes a presentation signal generating unit 424-n, a video presentation unit 421-n, and an audio presentation unit 422-n, as illustrated in Fig. 11. The non-haptic-sense-capable receiving device 420-n receives a signal SD for a non-haptic-sense-capable terminal output by the multipoint conference control device 101 connected via each transmission path 500-n. The non-haptic-sense-capable receiving device 420-n performs the processes of steps S424, S421, and S422 illustrated in Fig. 16 for each predetermined time period.
[0065] [Presentation signal generation unit 424-n] The presentation signal generation unit 424-n receives the signal SD for the non-haptic terminal. The presentation signal generation unit 424-n obtains a video signal from a video bit string, which is a bit string of a predetermined video transmission method included in the signal SD for the non-haptic terminal, and outputs the video signal to the video presentation unit 421-n, and obtains a signal of a channel for an audio signal from an audio bit string, which is a bit string of a predetermined audio transmission method included in the signal SD for the non-haptic terminal, and outputs the signal as an audio signal to the audio presentation unit 422-n (step S424).
[0066] The haptic non-compatible terminal device 400-n may be a terminal device that reproduces a multi-channel audio signal according to a predetermined audio transmission method. When the haptic non-compatible terminal device 400-n is a terminal device that reproduces a multi-channel audio signal according to a predetermined audio transmission method, the presentation signal generating unit 424-n obtains a channel signal for an audio signal from an audio bit string that is a bit string of a predetermined audio transmission method included in the haptic non-compatible terminal signal SD and outputs the signal as an audio signal to the audio presentation unit 422-n, and obtains a channel signal for a vibration signal from an audio bit string that is a bit string of a predetermined audio transmission method included in the haptic non-compatible terminal signal SD and outputs the signal as an audio signal to the audio presentation unit 422-n, in step S424.
[0067] [Video presentation section 421-n] The video presentation unit 421-n is, for example, a display device built into a PC or a display device connected to a PC. The video presentation unit 421-n receives the video signal output by the presentation signal generation unit 424-n. The video presentation unit 421-n displays a video corresponding to the input video signal, that is, presents the video (step S421).
[0068] [Sound presentation section 422-n] The audio presentation unit 422-n is, for example, a speaker built into the PC or a speaker connected to the PC. The audio presentation unit 422-n receives an audio signal output by the presentation signal generation unit 424-n. The audio presentation unit 422-n generates a sound corresponding to the input audio signal, that is, presents the sound (step S422).
[0069] Even when the haptic non-compatible terminal device 400-n is a terminal device that reproduces audio signals of multiple channels according to a predetermined audio transmission method, the audio presentation unit 422-n generates a sound corresponding to the input audio signal in step S422. That is, in this case, the audio presentation unit 422-n generates a sound corresponding to the signal of the channel for the vibration signal in addition to the sound corresponding to the signal of the channel for the audio signal included in the audio bit string. However, in the channel for the vibration signal, the conference signal generation unit 111 of the multipoint conference control device 101 stores an all-zero signal or an audio signal based on the audio signal input from each point. Therefore, even when the haptic non-compatible terminal device 400-n is a terminal device that reproduces audio signals of multiple channels according to a predetermined audio transmission method, the speaker does not emit an unexpected sound, and there is no possibility that the speaker will break down or be destroyed.
[0070] [Whether or not video is shown] It is not essential that the non-haptic-enabled receiving device 420-n presents an image. For example, the non-haptic-enabled receiving device 420-n may not include the image presenting unit 421-n, the presentation signal generating unit 424-n may not obtain an image signal from the signal SD for the non-haptic terminal, and the non-haptic-enabled receiving device 420-n may not perform step S421. Also, for example, the user may be able to select whether the non-haptic-enabled receiving device 420-n presents an image. That is, the non-haptic-enabled receiving device 420-n may be provided with a presentation signal selecting unit 425-n including an image presentation ON / OFF switch that is a switch that accepts an input by the user regarding whether the non-haptic-enabled receiving device 420-n presents an image, and the presentation signal generating unit 424-n may operate to output or not output a video signal according to the input accepted by the image presentation ON / OFF switch of the presentation signal selecting unit 425-n.
[0071] [Whether or not sound is presented] It is not essential that the non-haptic-enabled receiving device 420-n presents sound. For example, the non-haptic-enabled receiving device 420-n may not include the audio presentation unit 422-n, the presentation signal generation unit 424-n may not obtain an audio signal from the signal SD for the non-haptic-enabled terminal, and the non-haptic-enabled receiving device 420-n may not perform step S422. Also, for example, the user may be able to select whether the non-haptic-enabled receiving device 420-n presents sound. That is, the non-haptic-enabled receiving device 420-n may be provided with a presentation signal selection unit 425-n including an audio presentation ON / OFF switch that is a switch that accepts an input by the user regarding whether the non-haptic-enabled receiving device 420-n presents sound, and the presentation signal generation unit 424-n may operate to output or not output an audio signal according to the input accepted by the audio presentation ON / OFF switch of the presentation signal selection unit 425-n.
[0072] <Modification 1 of the First Embodiment> In order for the multipoint conference control device 101 to perform the above-mentioned operations, the multipoint conference control device 101 needs to be able to identify whether each terminal device connected via each transmission path is a tactile-enabled terminal device or a tactile-incompatible terminal device. The multipoint conference control device 101 may identify whether each terminal device is a tactile-enabled terminal device or a tactile-incompatible terminal device based on identification information or initial settings in a connection via the Internet, or may identify based on information output by each terminal device. An example in which the multipoint conference control device 101 identifies whether each terminal device is a tactile-enabled terminal device or a tactile-incompatible terminal device based on information output by each terminal device will be described as Modification 1 of the first embodiment, with a difference from the first embodiment.
[0073] [Haptic-enabled receiving device 220-m, supported media information transmitting unit 226-m] The haptic-enabled receiving device 220-m also includes a supported media information transmitter 226-m, as indicated by a dashed line in Fig. 2. The supported media information transmitter 226-m outputs information indicating that the terminal device m is a haptic-enabled terminal device, i.e., information indicating that the receiving device m is a receiving device capable of presenting a vibration signal, as supported media information Im to the multipoint conference control device 101 (step S226).
[0074] [Haptic incompatible receiving device 420-n, compatible media information transmitting unit 426-n] The tactile incompatible receiving device 420-n also includes a compatible media information transmitter 426-n, as indicated by a dashed line in Fig. 11. The compatible media information transmitter 426-n outputs information indicating that the terminal device n is a tactile incompatible terminal device, that is, information indicating that the receiving device n is a terminal device that cannot present a vibration signal, as compatible media information In to the multipoint conference control device 101 (step S426).
[0075] [Multipoint conference control device 101, conference signal generation unit 111] The multipoint conference control device 101 also receives input of compatible media information Ik (k is an integer between 1 and M+N) output by each terminal device connected via each transmission path. At the time when the compatible media information is input to the multipoint conference control device 101, the multipoint conference control device 101 cannot determine whether each terminal device is a tactile-compatible terminal device or a tactile-incompatible terminal device, so the compatible media information is represented here as Ik rather than Im and In. The compatible media information Ik input to the multipoint conference control device 101 is input to a conference signal generation unit 111. For each k between 1 and M+N, if the compatible media information Ik is information indicating that terminal device k is a tactile-compatible terminal device, i.e., information indicating that the receiving device k is a receiving device capable of presenting a vibration signal, the conference signal generating unit 111 identifies terminal device k as a tactile-compatible terminal device and identifies the terminal transmission signal input from terminal device k via the transmission path as a tactile-compatible terminal transmission signal; if the compatible media information Ik is information indicating that terminal device k is a tactile-incompatible terminal device, i.e., information indicating that the receiving device k is a receiving device that cannot present a vibration signal, the conference signal generating unit 111 identifies terminal device k as a tactile-incompatible terminal device and identifies the terminal transmission signal input from terminal device k via the transmission path as a tactile-incompatible terminal transmission signal (step S111-0), and then performs the processing of step S111 described in the first embodiment.
[0076] <Second embodiment> In a multipoint conference system in which a channel originally prepared for an audio signal in a predetermined audio transmission method is substituted for a channel for a vibration signal, a user of each terminal device may be allowed to select which of sound and vibration he or she wishes to reproduce, and the multipoint conference control device may generate a terminal-directed signal corresponding to the selection in each terminal device and output it to each terminal device. This embodiment is taken as the second embodiment, and the multipoint conference system of the second embodiment and each device included in the system will be described, focusing on the differences from the reference embodiment.
[0077] As shown in Fig. 1, the multipoint conference system 12 of the second embodiment includes a multipoint conference control device 102 and terminal devices 200-m. In the second embodiment, M is an integer of 2 or more, and m is an integer of 1 to M. The multipoint conference control device 102 and each terminal device 200-m are connected via each transmission path 300-m of a communication network. The communication network is, for example, the Internet, and is capable of transmitting signals between a plurality of devices mutually in almost real time. In the following, the point where the terminal device 200-m is located is referred to as point m.
[0078] [Terminal device 200-m] The terminal device 200-m is a terminal device located at a point m, and is configured, for example, by a PC and peripheral devices connected to the PC, and includes a transmitting device 210-m and a receiving device 220-m as shown in FIG.
[0079] [Transmitting device 210-m] The transmitting device 210-m is the same as the transmitting device 210-m in the reference embodiment, except that the transmitting device 210-m outputs a terminal sending signal SA-m to the multipoint conference control device 102 via a transmission path 300-m.
[0080] [Receiving device 220-m] 2, the receiving device 220-m includes a presentation signal generating unit 224-m, a video presenting unit 221-m, an audio presenting unit 222-m, a vibration presenting unit 223-m, a presentation signal selecting unit 225-m, and a corresponding media information transmitting unit 226-m. Here, the presentation signal selecting unit 225-m and the corresponding media information transmitting unit 226-m will be described first. The presentation signal generating unit 224-m, the video presenting unit 221-m, the audio presenting unit 222-m, and the vibration presenting unit 223-m will be described after the multipoint conference control device 102 is described.
[0081] [Presentation signal selection unit 225] The presentation signal selection unit 225 accepts a user's selection of any of the options including at least three options for sound and vibration: the receiving device 220-m presents both sound and vibration, the receiving device 220-m presents sound without presenting vibration, and the receiving device 220-m presents vibration without presenting sound, and outputs information about the selected option to the corresponding media information transmission unit 226-m. Of course, the options may also include the receiving device 220-m presenting neither sound nor vibration. In the above example using a two-channel stereo sound transmission method, one of the two channels is a channel for sound signals, and one of the two channels different from the channel for sound signals is a channel for vibration signals, and there is one channel for sound signals and one channel for vibration signals, so that the options may include only three options: the receiving device 220-m presents both sound and vibration, the receiving device 220-m presents sound without presenting vibration, and the receiving device 220-m presents vibration without presenting sound. In the above-mentioned example using a 5.1 channel surround sound transmission method, two of the six channels are for sound signals, and four of the six channels other than the channels for sound signals are for vibration signals, and there are multiple channels for sound signals and vibration signals, so the options include the three options of the receiving device 220-m presenting both sound and vibration, the receiving device 220-m presenting sound without presenting vibration, and the receiving device 220-m presenting vibration without presenting sound, and may also include which of the two channels for sound signals to present when the receiving device 220-m presents sound, and which of the four channels for vibration signals to present when the receiving device 220-m presents vibration.
[0082] The presentation signal selection unit 225 may also allow the user to select whether the receiving device 220-m presents video. That is, the presentation signal selection unit 225 may also include a video presentation ON / OFF switch that receives an input from the user regarding whether the receiving device 220-m presents video, and may also output information received by the video presentation ON / OFF switch to the compatible media information transmission unit 226-m.
[0083] [Supported media information transmission unit 226-m] The supported media information transmitter 226-m outputs supported media information Im including information corresponding to the above-mentioned options for sound and vibration input from the presentation signal selector 225 to the multipoint conference control device 102. In other words, when there are three options, that is, that the receiving device 220-m presents both sound and vibration, that the receiving device 220-m presents sound without presenting vibration, and that the receiving device 220-m presents vibration without presenting sound, the supported media information transmitter 226-m outputs supported media information Im including any of information indicating that "the receiving device 220-m presents both sound and vibration", information indicating that "the receiving device 220-m presents sound without presenting vibration", and information indicating that "the receiving device 220-m presents vibration without presenting sound". In addition, if there are four options, including the option that the receiving device 220-m does not present either sound or vibration, the compatible media information transmission unit 226-m outputs compatible media information Im including any of the four pieces of information, which are the above-mentioned three pieces of information plus information stating that "the receiving device 220-m does not present either sound or vibration."
[0084] When the information accepted by the video presentation ON / OFF switch is also input from the presentation signal selection unit 225, the compatible media information transmission unit 226-m outputs compatible media information Im including the information accepted by the video presentation ON / OFF switch to the multipoint conference control unit 102. In other words, the compatible media information transmission unit 226-m outputs compatible media information Im including either information indicating that "the receiving side device 220-m presents video" or information indicating that "the receiving side device 220-m does not present video" to the multipoint conference control unit 102.
[0085] The supported media information transmitter 226-m needs to output the supported media information Im at least when the terminal device 200-m is connected to the multipoint conference control device 102 and when the supported media information Im of the receiving device 220-m changes, but may also output the supported media information Im at other times. For example, the supported media information transmitter 226-m may output the supported media information Im at every predetermined time interval, or may output the supported media information Im at time intervals longer than the predetermined time interval.
[0086] [Multipoint conference control device 102] The multipoint conference control device 102 includes a conference signal generating unit 112, as shown in Fig. 17. The multipoint conference control device 102 receives corresponding media information I-1 to IM output by each of the terminal devices 200-1 to 200-M connected via each of the transmission paths 300-1 to 300-M, and terminal transmission signals SA-1 to SA-M output by each of the terminal devices 200-1 to 200-M connected via each of the transmission paths 300-1 to 300-M. The multipoint conference control device 102 performs the process of step S112 shown in Fig. 18 for each predetermined time period.
[0087] [Conference signal generator 112] Corresponding media information I-1 to IM and terminal transmission signals SA-1 to SA-M are input to the conference signal generation unit 112. The conference signal generation unit 112 performs each process of steps S112-1 to S112-7 in Fig. 19, for example, to obtain and output terminal-directed signals SE-1 to SE-M (step S112).
[0088] The conference signal generating unit 112 generates a video signal for output including a video signal included in the video bit string of each point included in each of the terminal sending signals SA-1 to SA-M (step S112-1). For example, similar to the multipoint conference system of Non-Patent Document 1, the conference signal generating unit 112 may obtain a reduced video by reducing the size of the video of each point, generate one video including the reduced video of each point, and generate a video signal corresponding to the generated video. When the terminal sending signals SA-1 to SA-M include a terminal sending signal that does not include a video bit string, the conference signal generating unit 112 may generate a video signal for output including a video signal of each point only from the terminal sending signals including a video bit string. After performing step S112-1, the conference signal generating unit 112 generates a video bit string that is a bit string of a predetermined video transmission method including the video signal obtained in step S112-1 (step S112-2). After performing step S112-2, the conference signal generating unit 112 also obtains a terminal-directed signal SE-G including the video bitstream generated in step S112-2 (step S112-6G).
[0089] The conference signal generating unit 112 generates an output acoustic signal including an acoustic signal included in the acoustic vibration bit string of each point included in each of the terminal transmission signals SA-1 to SA-M (step S112-3). The conference signal generating unit 112 may generate one acoustic signal by mixing the acoustic signals of each point, as in the multipoint conference system of Non-Patent Document 1, for example. When there is a terminal transmission signal in which an acoustic signal is not included in the acoustic vibration bit string of the terminal transmission signals SA-1 to SA-M, the conference signal generating unit 112 may generate an output acoustic signal including an acoustic signal of each point only for the terminal transmission signal in which an acoustic signal is included in the acoustic vibration bit string. When there are multiple channels for acoustic signals, the conference signal generating unit 112 may generate an output acoustic signal including an acoustic signal of each point for each channel.
[0090] Furthermore, the conference signal generating unit 112 generates an output vibration signal including a vibration signal included in the acoustic vibration bit string of each point included in each of the terminal sending signals SA-1 to SA-M (step S112-4). The conference signal generating unit 112 may generate one vibration signal by, for example, mixing the vibration signals of each point. When there is a terminal sending signal in which the acoustic vibration bit string of the terminal sending signals SA-1 to SA-M does not include a vibration signal, the conference signal generating unit 112 may generate an output vibration signal including a vibration signal of each point only for the terminal sending signal in which the acoustic vibration bit string includes a vibration signal. When there are multiple channels for vibration signals, the conference signal generating unit 112 may generate an output vibration signal including a vibration signal of each point for each channel.
[0091] After performing steps S112-3 and S111-4, the conference signal generating unit 112 stores the acoustic signal obtained in step S112-3 in a predetermined channel for acoustic signals among the multiple channels of the predetermined acoustic transmission method, and stores the vibration signal obtained in step S112-4 in a predetermined channel for vibration signals different from the channel for acoustic signals among the multiple channels of the predetermined acoustic transmission method, thereby generating an acoustic vibration bit string that is a bit string of the predetermined acoustic transmission method corresponding to the multiple channels (step S112-5A). After performing steps S112-2 and S112-5A, the conference signal generating unit 112 obtains a terminal-directed signal SE-A including the video bit string generated in step S112-2 and the acoustic vibration bit string generated in step S112-5A (step S112-6A). After performing step S112-5A, the conference signal generating unit 112 obtains a terminal-directed signal SE-D including the acoustic vibration bit string generated in step S112-5A (step S112-6D).
[0092] After performing step S112-3, the conference signal generating unit 112 stores the acoustic signal obtained in step S112-3 in a predetermined channel for acoustic signals among the multiple channels of the predetermined acoustic transmission method, and stores an all-zero signal or the acoustic signal obtained in step S112-3 in a predetermined channel for vibration signals different from the channel for acoustic signals among the multiple channels of the predetermined acoustic transmission method to generate an acoustic bit string that is a bit string of the predetermined acoustic transmission method corresponding to the multiple channels (step S112-5B). After performing steps S112-2 and S112-5B, the conference signal generating unit 112 obtains a terminal-directed signal SE-B including the video bit string generated in step S112-2 and the acoustic bit string generated in step S112-5B (step S112-6B). After performing step S112-5B, the conference signal generating unit 112 obtains a terminal-directed signal SE-E including the acoustic bit string generated in step S112-5B (step S112-6E). In addition, when storing an acoustic signal in a channel for vibration signals, if there are multiple predetermined channels for acoustic signals, the conference signal generating unit 112 may store the acoustic signal of any one of the channels in the channel for vibration signals, or may store a mixture of the acoustic signals of multiple channels in the channel for vibration signals.
[0093] After performing step S112-4, the conference signal generating unit 112 stores the all-zero signal or the vibration signal obtained in step S112-4 in a predetermined channel for the acoustic signal among the multiple channels of the predetermined acoustic transmission method, and stores the vibration signal obtained in step S112-4 in a predetermined channel for the vibration signal different from the channel for the acoustic signal among the multiple channels of the predetermined acoustic transmission method, thereby generating a vibration bit string that is a bit string of the predetermined acoustic transmission method corresponding to the multiple channels (step S112-5C). After performing steps S112-2 and S112-5C, the conference signal generating unit 112 obtains a terminal-directed signal SE-C including the video bit string generated in step S112-2 and the vibration bit string generated in step S112-5C (step S112-6C). After performing step S112-5C, the conference signal generating unit 112 obtains a terminal-directed signal SE-F including the vibration bit string generated in step S112-5C (step S112-6F). In addition, when storing a vibration signal in a channel for an acoustic signal, if there are multiple predetermined channels for vibration signals, the conference signal generating unit 112 may store the vibration signal of any one of the channels in the channel for the acoustic signal, or may store a mixture of the vibration signals of the multiple channels in the channel for the acoustic signal.
[0094] The conference signal generating unit 112 also obtains a terminal-directed signal SE-H that does not include a video bit string, an acoustic vibration bit string, an audio bit string, or a vibration bit string (step S112-6H). Even if the terminal-directed signal SE-H does not include a video bit string or an acoustic vibration bit string, the conference signal generating unit 112 needs to obtain the terminal-directed signal SE-H if the terminal-directed signal SE-H includes information that is not a video, sound, or vibration. However, if there is no information that needs to be included in and sent to the terminal-directed signal SE-H, the conference signal generating unit 112 does not need to obtain the terminal-directed signal SE-H and does not need to perform the process described later, i.e., the process of outputting the terminal-directed signal SE-H to the terminal device as the terminal-directed signal SE-m.
[0095] After performing steps S112-6A to S112-6H, the conference signal generating unit 112 obtains signals SE-1 to SE-M for each terminal in one of the following eight ways (A) to (H) and outputs them to each terminal device, i.e., outputs them to each of the terminal devices 200-1 to 200-M via each of the transmission paths 300-1 to 300-M (step S112-7).
[0096] (A) When the corresponding media information Im includes information indicating that “the receiving device 220-m presents both sound and vibration” and information indicating that “the receiving device 220-m presents video”, the conference signal generating unit 112 sets the terminal-directed signal SE-A to the terminal-directed signal SE-m. (B) When the compatible media information Im includes information indicating that “the receiving device 220-m presents sound without presenting vibration” and information indicating that “the receiving device 220-m presents video”, the conference signal generating unit 112 sets the terminal-directed signal SE-B to the terminal-directed signal SE-m. (C) When the compatible media information Im includes information indicating that “the receiving device 220-m presents vibration without presenting sound” and information indicating that “the receiving device 220-m presents video”, the conference signal generating unit 112 sets the terminal-directed signal SE-C to the terminal-directed signal SE-m. (D) When the compatible media information Im includes information indicating that “the receiving device 220-m presents both sound and vibration” and information indicating that “the receiving device 220-m does not present video”, the conference signal generating unit 112 sets the terminal-directed signal SE-D to the terminal-directed signal SE-m. (E) When the compatible media information Im includes information indicating that “the receiving device 220-m presents sound without presenting vibration” and information indicating that “the receiving device 220-m does not present video”, the conference signal generating unit 112 sets the terminal-directed signal SE-E to the terminal-directed signal SE-m. (F) If the compatible media information Im includes information indicating that "the receiving device 220-m presents vibration but not sound" and information indicating that "the receiving device 220-m does not present video", the conference signal generating unit 112 sets the terminal-directed signal SE-F to the terminal-directed signal SE-m. (G) When the compatible media information Im includes information indicating that “the receiving device 220-m presents neither sound nor vibration” and information indicating that “the receiving device 220-m presents video”, the conference signal generating unit 112 sets the terminal-directed signal SE-G to the terminal-directed signal SE-m. (H) When the compatible media information Im includes information indicating that “the receiving device 220-m does not present sound or vibration” and information indicating that “the receiving device 220-m does not present video”, the conference signal generating unit 112 sets the terminal-directed signal SE-H to the terminal-directed signal SE-m.
[0097] If none of the supported media information I-1 to IM applies among the above (A) to (H), the conference signal generating unit 112 does not need to generate the terminal-directed signals SE-A to SE-H that do not apply to any of the supported media information I-1 to IM, and in this case, the conference signal generating unit 112 does not need to perform unnecessary steps among steps S112-6A to S112-6H and the steps before them. Given that the terminal-directed signal that the conference signal generating unit 112 actually outputs is SE-m, the terminal-directed signals SE-A to SE-H are provisional terminal-directed signals in the conference signal generating unit 112, and the terminal-directed signals SE-A to SE-H can also be said to be "provisional terminal-directed signals SE-A to SE-H".
[0098] [The presentation signal generating unit 224-m, the video presentation unit 221-m, the audio presentation unit 222-m, and the vibration presentation unit 223-m of the receiving device 220-m] The presentation signal generating unit 224-m receives the terminal-directed signal SE-m. The presentation signal generating unit 224-m, the video presentation unit 221-m, the audio presentation unit 222-m, and the vibration presentation unit 223-m perform one of the following eight operations according to the selection received by the presentation signal selecting unit 225.
[0099] (A) When the selection received by the presentation signal selection unit 225 is a selection to present a sound, a vibration, and an image. The presentation signal generating unit 224-m obtains a video signal from a video bit string, which is a bit string of a predetermined video transmission method included in the terminal-directed signal SE-m, and outputs the video signal to the video presentation unit 221-m, obtains a signal of a channel for an audio signal from an audio vibration bit string, which is a bit string of a predetermined audio transmission method included in the terminal-directed signal SE-m, and outputs the signal as an audio signal to the audio presentation unit 222-m, and obtains a signal of a channel for a vibration signal from an audio vibration bit string, which is a bit string of a predetermined audio transmission method included in the terminal-directed signal SE-m, and outputs the signal as a vibration signal to the vibration presentation unit 223-m. The video presentation unit 221-m presents a video corresponding to the input video signal. The audio presentation unit 222-m presents a sound corresponding to the input audio signal. The vibration presentation unit 223-m presents a vibration corresponding to the input vibration signal.
[0100] (B) When the selection received by the presentation signal selection unit 225 is a selection to present a sound and an image without presenting a vibration. The presentation signal generating unit 224-m obtains a video signal from a video bit string, which is a bit string of a predetermined video transmission method included in the terminal-directed signal SE-m, and outputs it to the video presenting unit 221-m, and obtains a signal of a channel for an audio signal from an audio bit string, which is a bit string of a predetermined audio transmission method included in the terminal-directed signal SE-m, and outputs it as an audio signal to the audio presenting unit 222-m. The video presenting unit 221-m presents a video corresponding to the input video signal. The audio presenting unit 222-m presents a sound corresponding to the input audio signal. The presentation signal generating unit 224-m may obtain a signal of a channel for a vibration signal from the audio bit string and output it as an audio signal to the audio presenting unit 222-m. That is, the receiving side device 220-m may present a signal of a channel for a vibration signal included in the terminal-directed signal SE-m as a sound. Since no signal is input to the vibration presenting unit 223-m, the vibration presenting unit 223-m does not present a vibration.
[0101] (C) When the selection received by the presentation signal selection unit 225 is a selection to present a vibration and an image without presenting a sound. The presentation signal generating unit 224-m obtains a video signal from a video bit string, which is a bit string of a predetermined video transmission method included in the terminal-directed signal SE-m, and outputs it to the video presenting unit 221-m, and obtains a signal of a channel for a vibration signal from a vibration bit string, which is a bit string of a predetermined audio transmission method included in the terminal-directed signal SE-m, and outputs it as a vibration signal to the vibration presenting unit 223-m. The video presenting unit 221-m presents a video corresponding to the input video signal. The vibration presenting unit 223-m presents a vibration corresponding to the input vibration signal. The presentation signal generating unit 224-m may obtain a signal of a channel for an audio signal from the vibration bit string and output it as a vibration signal to the vibration presenting unit 223-m. That is, the receiving side device 220-m may present a signal of a channel for an audio signal included in the terminal-directed signal SE-m as a vibration. Since no signal is input to the audio presenting unit 222-m, the audio presenting unit 222-m does not present a sound.
[0102] (D) When the selection received by the presentation signal selection unit 225 is a selection to present a sound, a vibration, and no image. The presentation signal generating unit 224-m obtains a signal of a channel for an audio signal from an audio vibration bit string, which is a bit string of a predetermined audio transmission method included in the terminal-directed signal SE-m, and outputs the signal as an audio signal to the audio presentation unit 222-m, and obtains a signal of a channel for a vibration signal from an audio vibration bit string, which is a bit string of a predetermined audio transmission method included in the terminal-directed signal SE-m, and outputs the signal as a vibration signal to the vibration presentation unit 223-m. The audio presentation unit 222-m presents a sound corresponding to the input audio signal. The vibration presentation unit 223-m presents a vibration corresponding to the input vibration signal. Since no signal is input to the video presentation unit 221-m, the video presentation unit 221-m does not present an image.
[0103] (E) When the selection received by the presentation signal selection unit 225 is a selection to present a sound, not a vibration, and not an image. The presentation signal generating unit 224-m obtains a signal of a channel for an audio signal from an audio bit string, which is a bit string of a predetermined audio transmission method included in the terminal-directed signal SE-m, and outputs the signal as an audio signal to the audio presentation unit 222-m. The audio presentation unit 222-m presents a sound corresponding to the input audio signal. The presentation signal generating unit 224-m may obtain a signal of a channel for a vibration signal from the audio bit string and output the signal as an audio signal to the audio presentation unit 222-m. That is, the receiving device 220-m may present a signal of a channel for a vibration signal included in the terminal-directed signal SE-m as a sound. Since no signal is input to the video presentation unit 221-m, the video presentation unit 221-m does not present an image. Since no signal is input to the vibration presentation unit 223-m, the vibration presentation unit 223-m does not present a vibration.
[0104] (F) When the selection received by the presentation signal selection unit 225 is a selection to present vibration without presenting sound and without presenting video. The presentation signal generating unit 224-m obtains a signal of a channel for a vibration signal from a vibration bit string, which is a bit string of a predetermined sound transmission method included in the terminal-directed signal SE-m, and outputs the signal as a vibration signal to the vibration presentation unit 223-m. The vibration presentation unit 223-m presents a vibration corresponding to the input vibration signal. The presentation signal generating unit 224-m may obtain a signal of a channel for an acoustic signal from the vibration bit string and output the signal as a vibration signal to the vibration presentation unit 223-m. That is, the receiving side device 220-m may present a signal of a channel for an acoustic signal included in the terminal-directed signal SE-m as a vibration. Since no signal is input to the video presentation unit 221-m, the video presentation unit 221-m does not present a video. Since no signal is input to the audio presentation unit 222-m, the audio presentation unit 222-m does not present a sound.
[0105] (G) A case in which the selection accepted by the presentation signal selection unit 225 is a selection to present an image without presenting a sound or vibration. The presentation signal generation unit 224-m obtains a video signal from a video bit string, which is a bit string of a predetermined video transmission method included in the terminal-directed signal SE-m, and outputs the video signal to the video presentation unit 221-m. The video presentation unit 221-m presents a video corresponding to the input video signal. Since no signal is input to the audio presentation unit 222-m, the audio presentation unit 222-m does not present a sound. Since no signal is input to the vibration presentation unit 223-m, the vibration presentation unit 223-m does not present a vibration.
[0106] (H) When the selection received by the presentation signal selection unit 225 is a selection of not presenting sound, not presenting vibration, and not presenting video. The presentation signal generation unit 224-m does not output a video signal, an audio signal, or a vibration signal. Since no signal is input to the video presentation unit 221-m, the video presentation unit 221-m does not present a video. Since no signal is input to the audio presentation unit 222-m, the audio presentation unit 222-m does not present a sound. Since no signal is input to the vibration presentation unit 223-m, the vibration presentation unit 223-m does not present a vibration.
[0107] <Modifications 1 and 2 of the Second Embodiment> The presentation signal selection unit 225 may include, for example, a presentation signal selection user interface area that is displayed on the screen of the video presentation unit 221-m of the terminal device 200-m, accepts a user's input, and displays the result. The user's input may be accepted by a well-known means such as a mouse provided in the terminal device 200-m or a touch operation on a display device that is a touch screen. An example of a case where a 2-channel stereo sound transmission method is used will be described as Variation 1, and an example of a case where a 5.1-channel surround sound transmission method is used will be described as Variation 2.
[0108] <Modification 1 of the second embodiment> 20A to 20C, the screen 2211-m of the video presentation unit 221-m includes a video display area 2212-m for displaying video, and a presentation signal selection user interface area 2251-m. The presentation signal selection user interface area 2251-m includes a media group switching button group 2252-m with a "+" button and a "-" button for cyclically switching between FIG. 20A, which is a display of a state in which sound is presented without vibration, FIG. 20B, which is a display of a state in which both sound and vibration are presented, and FIG. 20C, which is a display of a state in which vibration is presented without sound, and a media display group 2253-m for displaying each medium presented in each state.
[0109] When the media group switching button group 2252-m receives a press of the “+” button while the screen 2211-m is displaying FIG. 20A, the presentation signal selection unit 225 causes the screen 2211-m to display FIG. 20B, when the presentation signal selection unit 225 receives a press of the “+” button while the screen 2211-m is displaying FIG. 20B, the presentation signal selection unit 225 causes the screen 2211-m to display FIG. 20C, and when the presentation signal selection unit 225 receives a press of the “+” button while the screen 2211-m is displaying FIG. 20C, the presentation signal selection unit 225 causes the screen 2211-m to display FIG. 20A. 20A, the presentation signal selection unit 225 causes screen 2211-m to display FIG. 20C, and when the "-" button is pressed while screen 2211-m is displaying FIG. 20C, the presentation signal selection unit 225 causes screen 2211-m to display FIG. 20B, and when the "-" button is pressed while screen 2211-m is displaying FIG. 20B, the presentation signal selection unit 225 causes screen 2211-m to display FIG. 20A.
[0110] The media group switching button group 2252-m consisting of the "+" button and the "-" button allows easy and cyclic switching between three possible combinations of sound and vibration presentation in a terminal device of a multipoint conference system in which a channel originally prepared for an audio signal in a specified audio transmission method is substituted as a channel for a vibration signal. The media group switching button group 2252-m may have only the above-mentioned "+" button and not the above-mentioned "-" button, or may have only the above-mentioned "-" button and not the above-mentioned "+" button. When the media group switching button group 2252-m has only one button, it is not a "group," and therefore can be said to be a media group switching button 2252-m.
[0111] The media display group 2253-m is for displaying the selection result by the media group switching button group 2252-m. For example, the media display group 2253-m in FIG. 20A including "video", "sound L", and "sound R" is a display that visually indicates a state in which video is presented, sound is presented, and vibration is not presented. The display of "sound L" and "sound R" in FIG. 20A means that the audio presentation unit 222-m presents sound in two channels. In this case, for example, the presentation signal generation unit 224-m obtains a signal of a channel for an audio signal from the audio vibration bit string and outputs it to the audio presentation unit 222-m as an audio signal for the left channel, and obtains a signal of a channel for a vibration signal from the audio vibration bit string and outputs it to the audio presentation unit 222-m as an audio signal for the right channel, and the audio presentation unit 222-m presents sounds corresponding to the audio signals of the two input channels. As described above, when the receiving device 220-m presents sound but not vibration, the signal in the channel for the vibration signal is either an all-zero signal or an acoustic signal that is the same as the signal in the channel for the acoustic signal, so there is no problem in presenting a sound corresponding to the signal in the channel for the vibration signal.
[0112] Also, for example, the media display group 2253-m in FIG. 20B including "video", "sound L", and "vibration 1" is a display that allows visual confirmation that the video, sound, and vibration are being presented. The display "sound L" in FIG. 20B means that the audio presentation unit 222-m presents one channel of sound, and in this case, the presentation signal generation unit 224-m obtains a signal for a channel for an audio signal from the audio vibration bit string and outputs it to the audio presentation unit 222-m as an audio signal, and the audio presentation unit 222-m presents a sound corresponding to the input audio signal of one channel. The display "vibration 1" in FIG. 20B means that the audio presentation unit 222-m presents one channel of vibration, and in this case, the presentation signal generation unit 224-m obtains a signal for a channel for a vibration signal from the audio vibration bit string and outputs it to the vibration presentation unit 223-m as a vibration signal, and the vibration presentation unit 223-m presents a vibration corresponding to the input vibration signal of one channel.
[0113] Also, for example, the media display group 2253-m in Fig. 20C including "video", "vibration 1" and "vibration 2" is a display that allows visual confirmation that a video is presented, a sound is not presented and a vibration is presented. The display of "vibration 1" and "vibration 2" in Fig. 20C means that the vibration presentation unit 223-m presents vibrations of two channels, and in this case, the presentation signal generation unit 224-m obtains a signal of a channel for a vibration signal from the acoustic vibration bit string and outputs it to the vibration presentation unit 222-m as a vibration signal of a first channel, and obtains a signal of a channel for an acoustic signal from the acoustic vibration bit string and outputs it to the vibration presentation unit 223-m as a vibration signal of a second channel, and the vibration presentation unit 223-m presents vibrations corresponding to the vibration signals of the two input channels. As described above, when the receiving device 220-m does not present sound but presents vibration, the signal in the channel for the acoustic signal is either an all-zero signal or the same vibration signal as the signal in the channel for the vibration signal, so there is no problem in presenting a vibration corresponding to the signal in the channel for the acoustic signal.
[0114] The media display group 2253-m may function as a button for switching whether or not to present each media or each channel in the selection result by the media group switching button group 2252-m. For example, the display of "Video" in the media display group 2253-m may function as a button for accepting an input by a user as to whether or not the receiving device 220-m presents a video. The same applies to "Sound L", "Sound R", "Vibration 1", and "Vibration 2". The displays of "Sound L" and "Sound R" in the media display group 2253-m may function as buttons for accepting an input by a user as to whether or not the receiving device 220-m presents a sound, and the displays of "Vibration 1" and "Vibration 2" in the media display group 2253-m may function as buttons for accepting an input by a user as to whether or not the receiving device 220-m presents a vibration. Also, for example, the displays of "sound L" and "sound R" in the media display group 2253-m may function as buttons for accepting input from the user as to whether or not the receiving device 220-m presents sound on each channel. Similarly, the displays of "vibration 1" and "vibration 2" in the media display group 2253-m may function as buttons for accepting input from the user as to whether or not the receiving device 220-m presents vibration on each channel. When the media display group 2253-m is made to function as buttons for switching whether or not each media or each channel is presented, it is preferable to display whether or not a video is presented, whether or not a sound is presented, and whether or not a vibration is presented by the color and / or brightness of each button. Note that when the media display group 2253-m functions as a button for switching whether or not each media or each channel is presented, it can be said to be a media ON / OFF button group 2253-m.
[0115] <Modification 2 of the second embodiment> The presentation signal selection user interface area 2251-m may include a media group switching button group 2252-m or a media group switching button 2252-m, and a media display group 2253-m or a media ON / OFF button group 2253-m so that it functions in the same way as variant example 1 when a 5.1 channel surround audio transmission method is used and when a 2 channel stereo audio transmission method is used.
[0116] For example, the media group switching button group 2252-m shown in Figures 21A to 21C allows easy and cyclic switching between three possible combinations of sound and vibration presentation and, when vibration is presented, which of the four vibration signal channels is to be presented. For example, when the media group switching button group 2252-m receives a press of the "+" button while the screen 2211-m is displaying FIG. 21A, it displays a screen with "Vibration 1" added to the media display group 2253-m of FIG. 21A, and when the "+" button is received while the screen is displaying the screen with "Vibration 1" added to the media display group 2253-m of FIG. 21A, it displays FIG. 21B. When the "+" button is received while the screen 2211-m is displaying FIG. 21B, it displays a screen with "Vibration 3" added to the media display group 2253-m of FIG. 21B, and when the "+" button is received while the screen is displaying the screen with "Vibration 3" added to the media display group 2253-m of FIG. 21B, it displays FIG. 21C. 21A to 21C are for displaying the selection results by the media group switching button group 2252-m, and are similar to the media display group 2253-m of Modification 1. That is, the media display group 2253-m may be made to function as a button for switching whether or not to present each media or each channel, to be the media ON / OFF button group 2253-m.
[0117] <Third embodiment> A multipoint conference system that functions similarly to the first embodiment can be realized by providing a signal conversion device between the multipoint conference control device and each terminal device, and using a multipoint conference control device that receives and outputs one type of terminal-oriented signal, similar to the multipoint conference system 10 of the reference embodiment. This embodiment will be referred to as the third embodiment, and the differences from the first embodiment will be mainly described.
[0118] As illustrated in FIG. 22, the multipoint conference system 13 of the third embodiment includes a multipoint conference control device 103, a tactile-enabled terminal device 200-m, a non-tactile-enabled terminal device 400-n, and a signal conversion device 603-n. In the third embodiment, M is an integer of 1 or more, m is an integer of 1 or more and M or less, N is an integer of 1 or more, and n is an integer of M+1 or more and M+N or less. The multipoint conference control device 103 and each tactile-enabled terminal device 200-m are connected via each transmission path 300-m of the communication network. The multipoint conference control device 103 and each signal conversion device 603-n are connected via each transmission path 501-n of the communication network. Each signal conversion device 603-n and each non-tactile-enabled terminal device 400-n are connected via each transmission path 502-n. Each transmission path 502-n does not necessarily have to be a transmission path of a communication network, and may be a transmission path such as a LAN that connects a signal conversion device 603-n and a tactile non-compatible terminal device 400-n arranged at the same point n, or may be a path that transmits a signal inside a device in which the signal conversion device 603-n and the tactile non-compatible terminal device 400-n are integrated. The multipoint conference control device 103 and each tactile non-compatible terminal device 500-n may be connected via each transmission path 500-n of a communication network, as shown by the dashed line in FIG.
[0119] [Tactile terminal device 200-m] The haptic-enabled terminal device 200-m is the same as the haptic-enabled terminal device 200-m of the first embodiment. However, the haptic-enabled transmitting device 210-m outputs a haptic-enabled terminal sending signal SA-m to the multipoint conference control device 103 via a transmission path 300-m. Also, a haptic-enabled terminal-directed signal SB is input to the haptic-enabled receiving device 220-n from the multipoint conference control device 103 via the transmission path 300-m.
[0120] [Non-tactile terminal device 400-n] The haptic non-compatible terminal device 400-n is the same as the haptic non-compatible terminal device 400-n of the first embodiment. However, the haptic non-compatible transmitting side device 410-n outputs a haptic non-compatible terminal sending signal SC-n to the multipoint conference control device 103 via a transmission path 502-n, a signal conversion device 603-n, and a transmission path 501-n, or outputs the signal to the multipoint conference control device 103 via a transmission path 500-n without via the signal conversion device 603-n. Also, a haptic non-compatible terminal-oriented signal SD is input to the haptic non-compatible receiving side device 420-n from the signal conversion device 603-n via the transmission path 502-n.
[0121] [Multipoint conference control device 103] The multipoint conference control device 103 includes a conference signal generation unit 113, as shown in Fig. 23. The multipoint conference control device 103 receives tactile-compatible terminal output signals SA-1 to SA-M output from the tactile-compatible terminal devices 200-1 to 200-M and non-tactile-compatible terminal output signals SC-(M+1) to SC-(M+N) output from the non-tactile-compatible terminal devices 400-(M+1) to 400-(M+N). The multipoint conference control device 103 obtains and outputs a tactile-compatible terminal-directed signal SB by performing the process of step S113 shown in Fig. 24 for each predetermined time period.
[0122] [Conference signal generator 113] The conference signal generating unit 113 receives input of haptic-enabled terminal sending signals SA-1 to SA-M and non-haptic-enabled terminal sending signals SC-(M+1) to SC-(M+N). The conference signal generating unit 113 obtains and outputs a haptic-enabled terminal-directed signal SB from the input haptic-enabled terminal sending signals SA-1 to SA-M and non-haptic-enabled terminal sending signals SC-(M+1) to SC-(M+N) (step S113). The process by which the conference signal generating unit 113 obtains a haptic-enabled terminal-directed signal SB from the input haptic-enabled terminal sending signals SA-1 to SA-M and non-haptic-enabled terminal sending signals SC-(M+1) to SC-(M+N) is the same as that of the conference signal generating unit 111 of the first embodiment. Moreover, the conference signal generating unit 113 does not obtain or output a non-haptic-enabled terminal-directed signal SD. That is, the conference signal generation unit 113 performs steps S111-1 to S111-6, which are the same as those of the conference signal generation unit 111 of the first embodiment, but does not perform steps S111-7 and S111-8 performed by the conference signal generation unit 111 of the first embodiment. Also, the conference signal generation unit 113 outputs the tactile-compatible terminal-oriented signal SB to each tactile-compatible terminal device and each signal conversion device. That is, the conference signal generation unit 113 outputs the tactile-compatible terminal-oriented signal SB to each of the tactile-compatible terminal devices 200-1 to 200-M via each of the transmission paths 300-1 to 300-M, and outputs the signal conversion devices 603-(M+1) to 603-(M+N) via each of the transmission paths 501-(M+1) to 501-(M+N).
[0123] [Signal conversion device 603-n] The signal converter 603-n includes a conference signal converter 613-n as shown in Fig. 25. The signal converter 603-n receives a signal SB for a tactile-sense-compatible terminal output from the multipoint conference control device 103 connected via the transmission path 501-n. The signal converter 603-n performs the process of step S613 shown in Fig. 26 for each predetermined time period.
[0124] The conference signal converter 613-n receives the haptic-compatible terminal-oriented signal SB as input. The conference signal converter 613-n performs the processes of steps S613-1 to S613-4 shown in Fig. 27 to obtain and output the haptic non-compatible terminal-oriented signal SD from the received haptic-compatible terminal-oriented signal SB (step S613).
[0125] The conference signal conversion unit 613-n extracts an acoustic signal included in the acoustic vibration bit string included in the haptic terminal-oriented signal SB (step S613-1). After performing step S613-1, the conference signal conversion unit 613-n stores the acoustic signal obtained in step 613-1 in a predetermined channel for acoustic signals among the multiple channels of a predetermined acoustic transmission method, and stores an all-zero signal or the acoustic signal obtained in step S613-1 in a predetermined channel for vibration signals different from the channel for acoustic signals among the multiple channels of the predetermined acoustic transmission method, thereby generating an acoustic bit string that is a bit string of a predetermined acoustic transmission method corresponding to the multiple channels (step S613-2).
[0126] The conference signal converter 613-n extracts the video bit string contained in the haptic-enabled terminal-oriented signal SB (step S613-3). After performing steps S613-2 and S613-3, the conference signal converter 613-n obtains a haptic-non-enabled terminal-oriented signal SD including the audio bit string obtained in step 613-2 and the video bit string obtained in step 613-3, and outputs the signal SD to the corresponding haptic-non-enabled terminal device, i.e., outputs the signal via the transmission path 502-n to the haptic-non-enabled terminal device 400-n connected via the transmission path 502-n (step S613-4).
[0127] Depending on the convenience of connection to the transmission line, as shown by the dashed line in FIG. 25, the signal conversion device 603-n may receive a tactile non-compatible terminal output signal SC-n from the tactile non-compatible terminal device 400-n connected via the transmission line 502-n, and output the input tactile non-compatible terminal output signal SC-n as is to the multipoint conference control device 103 connected via the transmission line 501-n. When signal conversion device 603-n does not input and output non-tactile-capable terminal transmission signal SC-n, signal conversion devices 603-(M+1) to 603-(M+N) all perform the same processing, so multipoint conference system 13 does not need to be equipped with signal conversion devices 603-(M+1) to 603-(M+N) corresponding to each of tactile-capable non-terminal devices 400-(M+1) to 400-(M+N), and may be equipped with one signal conversion device for multiple tactile-capable non-terminal devices, or may be equipped with only one signal conversion device.
[0128] <Fourth embodiment> A multipoint conference system that functions similarly to the second embodiment can be realized by providing a signal conversion device between the multipoint conference control device and each terminal device, and using a multipoint conference control device that receives and outputs one type of terminal-oriented signal, similar to the multipoint conference system 10 of the reference embodiment. This embodiment will be described as the fourth embodiment, focusing on the differences from the second embodiment and also using the differences and similarities with the reference embodiment as appropriate.
[0129] As illustrated in FIG. 28, the multipoint conference system 14 of the fourth embodiment includes a multipoint conference control device 100, a terminal device 200-m, and a signal conversion device 604-m. In the fourth embodiment, M is an integer of 2 or more, and m is an integer of 1 to M. The multipoint conference control device 100 and each signal conversion device 604-m are connected via each transmission path 301-m of the communication network. Each signal conversion device 604-m and each terminal device 200-m are connected via each transmission path 302-m. Each transmission path 302-m does not necessarily have to be a transmission path of the communication network, and may be a transmission path such as a LAN that connects the signal conversion device 604-m and the terminal device 200-m located at the same point m, or may be a path that transmits a signal inside a device in which the signal conversion device 604-m and the terminal device 200-m are integrated. The multipoint conference control device 100 and each terminal device 200-m may be connected via each transmission path 300-m of a communication network, as shown by a dashed line in FIG.
[0130] [Terminal device 200-m] The terminal device 200-m is the same as the terminal device 200-m of the second embodiment. However, the terminal device 200-m outputs a terminal sending signal SA-m to the multipoint conference control device 100 via the transmission path 302-m, the signal conversion device 604-m, and the transmission path 301-m, or outputs the terminal sending signal SA-m to the multipoint conference control device 100 via the transmission path 300-m without the signal conversion device 604-m. The terminal device 200-m also outputs corresponding media information Im to the signal conversion device 604-m via the transmission path 302-m. The terminal device 200-m also receives a terminal-directed signal SE-m from the signal conversion device 604-m via the transmission path 302-m.
[0131] [Multipoint conference control device 100] The multipoint conference control device 100 is the same as the multipoint conference control device 100 of the reference embodiment. However, the multipoint conference control device 100 outputs a terminal-directed signal SB to each of the signal conversion devices 604-1 to 604-M via each of the transmission paths 301-1 to 301-M. That is, in this embodiment, the "terminal-directed signal SB" is not a signal directed to a terminal device but a signal directed to a signal conversion device, and therefore can also be said to be a "signal SB directed to a signal conversion device."
[0132] [Signal conversion device 604-m] The signal conversion device 604-m includes a conference signal conversion unit 614-m as illustrated in Fig. 29. The signal conversion device 604-m receives the terminal-directed signal SB output by the multipoint conference control device 100 and the corresponding media information Im output by the terminal device 200-m. The signal conversion device 604-m obtains the terminal-directed signal SE-m by performing the process of step S614 illustrated in Fig. 18 for each predetermined time period, and outputs the signal to the terminal device 200-m, that is, outputs the signal to the terminal device 200-m via each transmission path 301-m.
[0133] The conference signal conversion unit 614-m receives the terminal-directed signal SB and corresponding media information Im as input. The conference signal conversion unit 614-m performs, for example, any of the processes in steps S614-A to S614-H described below to obtain and output a terminal-directed signal SE-m from the input terminal-directed signal SB (step S614).
[0134] (A) When the corresponding media information Im includes information indicating that “the receiving device 220-m presents both sound and vibration” and information indicating that “the receiving device 220-m presents video”, the conference signal conversion unit 614-m outputs the terminal-directed signal SB as is as the terminal-directed signal SE-m (step S614-A).
[0135] (B) When the compatible media information Im includes information indicating that “the receiving device 220-m presents sound without presenting vibration” and information indicating that “the receiving device 220-m presents video”, the conference signal conversion unit 614-m performs steps S614-B1 to S614-B3 described below to obtain and output a terminal-directed signal SE-m (step S614-B).
[0136] The conference signal conversion unit 614-m obtains an acoustic signal, which is a signal of a channel for an acoustic signal, from an acoustic vibration bit string included in the terminal-directed signal SB (step S614-B1). After performing step S614-B1, the conference signal conversion unit 614-m stores the acoustic signal obtained in step S614-B1 in a channel for a predetermined acoustic signal among the multiple channels of a predetermined acoustic transmission method, and generates an acoustic bit string, which is a bit string of a predetermined acoustic transmission method corresponding to the multiple channels, by storing an all-zero signal or the acoustic signal obtained in step S614-B1 in a channel for a predetermined vibration signal different from the channel for an acoustic signal among the multiple channels of the predetermined acoustic transmission method (step S614-B2). After performing step S614-B2, the conference signal conversion unit 614-m obtains and outputs a terminal-directed signal SE-m including a video bit string included in the terminal-directed signal SB and the acoustic bit string obtained in step S614-B2 (step S614-B3).
[0137] (C) When the compatible media information Im includes information indicating that “the receiving device 220-m presents vibration but not sound” and information indicating that “the receiving device 220-m presents video”, the conference signal conversion unit 614-m performs steps S614-C1 to S614-C3 described below to obtain and output a terminal-directed signal SE-m (step S614-C).
[0138] The conference signal conversion unit 614-m obtains a vibration signal, which is a signal of a channel for vibration signals, from the acoustic vibration bit string included in the terminal-directed signal SB (step S614-C1). After performing step S614-C1, the conference signal conversion unit 614-m stores an all-zero signal or the vibration signal obtained in step S614-C1 in a predetermined channel for acoustic signals among the multiple channels of a predetermined acoustic transmission method, and stores the vibration signal obtained in step S614-C1 in a predetermined channel for vibration signals different from the channel for acoustic signals among the multiple channels of the predetermined acoustic transmission method, thereby generating a vibration bit string, which is a bit string of a predetermined acoustic transmission method corresponding to the multiple channels (step S614-C2). After performing step S614-C2, the conference signal conversion unit 614-1 obtains and outputs a terminal-directed signal SE-m including the video bit string included in the terminal-directed signal SB and the vibration bit string obtained in step S614-C2 (step S614-C3).
[0139] (D) When the compatible media information Im includes information indicating that "the receiving device 220-m presents both sound and vibration" and information indicating that "the receiving device 220-m does not present video", the conference signal conversion unit 614-m obtains and outputs a terminal-directed signal SE-m that does not include a video bitstream but includes the acoustic vibration bitstream contained in the terminal-directed signal SB (step S614-D).
[0140] (E) If the compatible media information Im includes information that "the receiving device 220-m presents sound without presenting vibration" and information that "the receiving device 220-m does not present video", the conference signal conversion unit 614-m performs the above-mentioned steps S614-B1 and S614-B2, and after performing step 614-B2, obtains and outputs a terminal-directed signal SE-m that does not include a video bitstream but includes the audio bitstream obtained in step S614-B2 (step S614-E).
[0141] (F) If the compatible media information Im includes information that "the receiving device 220-m presents vibration but not sound" and information that "the receiving device 220-m does not present video", the conference signal conversion unit 614-m performs the above-mentioned steps S614-C1 and S614-C2, and after performing step 614-C2, obtains and outputs a terminal-directed signal SE-m that does not include a video bitstream but includes the vibration bitstream obtained in step S614-C2 (step S614-F).
[0142] (G) When the compatible media information Im includes information indicating that "the receiving device 220-m presents neither sound nor vibration" and information indicating that "the receiving device 220-m presents video", the conference signal conversion unit 614-m obtains and outputs a terminal-directed signal SE-m that does not include an acoustic vibration bit string, nor an audio bit string, nor a vibration bit string, but includes a video bit string included in the terminal-directed signal SB (step S614-G).
[0143] (H) When the compatible media information Im includes information that "the receiving device 220-m does not present sound or vibration" and information that "the receiving device 220-m does not present video", the conference signal conversion unit 614-m obtains and outputs a terminal-directed signal SE-m that does not include a video bit string, an audio vibration bit string, an audio bit string, or a vibration bit string (step S614-H). Note that even if the terminal-directed signal SE-m does not include a video bit string, an audio vibration bit string, an audio bit string, or a vibration bit string, the conference signal conversion unit 614-m needs to obtain and output the terminal-directed signal SE-m if the terminal-directed signal SE-m includes information that is not video, sound, or vibration, but if there is no information that needs to be included in and sent in the terminal-directed signal SE-m, it is not necessary to obtain and output the terminal-directed signal SE-m.
[0144] <Programs and recording media> The processing of each of the above-mentioned multipoint conference control devices, each terminal device, and each signal conversion device may be realized by a computer, in which case the processing contents of the functions that each device should have are described by a program. Then, by loading this program into the storage unit 1020 of the computer 1000 shown in Fig. 30 and operating the arithmetic processing unit 1010, input unit 1030, output unit 1040, etc., various processing functions of the above-mentioned devices are realized on the computer.
[0145] The program describing the processing contents can be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a non-transitory recording medium, specifically, a magnetic recording device, an optical disk, etc.
[0146] The program may be distributed, for example, by selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of a server computer and transferring the program from the server computer to other computers via a network.
[0147] A computer that executes such a program, for example, first stores a program recorded on a portable recording medium or a program transferred from a server computer in the auxiliary recording unit 1050, which is a non-transient storage device of the computer. Then, when executing a process, the computer reads the program stored in the auxiliary recording unit 1050, which is a non-transient storage device of the computer, into the storage unit 1020, and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from the portable recording medium into the storage unit 1020 and execute a process according to the program, or may execute a process according to the received program each time a program is transferred from the server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may execute the above-mentioned process by a so-called ASP (Application Service Provider) type service that realizes a processing function only by issuing an execution instruction and obtaining a result. Note that the program in this embodiment includes information used for processing by an electronic computer and equivalent to a program (data that is not a direct instruction to a computer but has a nature that specifies the processing of the computer, etc.).
[0148] In addition, in this embodiment, the present device is configured by executing a specific program on a computer, but at least a part of the processing contents may be realized by hardware.
[0149] Needless to say, other modifications can be made without departing from the spirit of the present invention.
Claims
1. A multipoint conference control device included in a multipoint conference system including a plurality of terminal devices and a multipoint conference control device connected to each of the terminal devices via a communication path, the multipoint conference system uses one or more channels of a plurality of channels of a predetermined acoustic transmission method as channels for acoustic signals, and uses one or more channels of the plurality of channels of the acoustic transmission method different from the channels for the acoustic signals as channels for vibration signals, a conference signal generating unit for obtaining a signal of the acoustic transmission method, in which, as a signal for a terminal device among the terminal devices that can present sound but cannot present vibration, an acoustic signal input from the terminal device via the communication path is set as a signal for the channel for the acoustic signal, and no signal or an acoustic signal input from the terminal device via the communication path is set as a signal for the channel for the vibration signal. A multipoint conference control device comprising:
2. A multipoint conference control device included in a multipoint conference system including a plurality of terminal devices and a multipoint conference control device connected to each of the terminal devices via a communication path, the multipoint conference system uses one or more channels of a plurality of channels of a predetermined acoustic transmission method as channels for acoustic signals, and uses one or more channels of the plurality of channels of the acoustic transmission method different from the channels for the acoustic signals as channels for vibration signals, a conference signal generating unit for obtaining a signal of the acoustic transmission method, the conference signal generating unit generating a signal for a terminal device among the terminal devices that can present vibration but cannot present sound, the signal being a vibration signal input from the terminal device through the communication path as a signal for the channel for the vibration signal, and a signal being no signal or a vibration signal input from the terminal device through the communication path as a signal for the channel for the acoustic signal. A multipoint conference control device comprising:
3. A multipoint conference control device included in a multipoint conference system including a plurality of terminal devices and a multipoint conference control device connected to each of the terminal devices via a communication path, the multipoint conference system uses one or more channels of a plurality of channels of a predetermined acoustic transmission method as channels for acoustic signals, and uses one or more channels of the plurality of channels of the acoustic transmission method different from the channels for the acoustic signals as channels for vibration signals, a signal of the acoustic transmission method is obtained, as a signal for a terminal device among the terminal devices that can present sound but cannot present vibration, in which an acoustic signal input from the terminal device via the communication path is set as a signal for the channel for the acoustic signal, and no signal or an acoustic signal input from the terminal device via the communication path is set as a signal for the channel for the vibration signal; a signal of the acoustic transmission method is obtained, as a signal for a terminal device among the terminal devices that can present vibration but cannot present sound, in which a vibration signal input from the terminal device through the communication path is set as a signal of a channel for the vibration signal, and no signal or a vibration signal input from the terminal device through the communication path is set as a signal of the channel for the acoustic signal; a conference signal generating unit for obtaining signals of the acoustic transmission method, the conference signal generating unit generating signals for a terminal device capable of presenting both sound and vibration among the terminal devices, the audio signal input from the terminal device via the communication path being a signal for the channel for the audio signal, and the vibration signal input from the terminal device via the communication path being a signal for the channel for the vibration signal. A multipoint conference control device comprising:
4. A multipoint conference control device included in a multipoint conference system including a plurality of terminal devices and a multipoint conference control device connected to each of the terminal devices via a communication path, the multipoint conference system uses one or more channels of a plurality of channels of a predetermined acoustic transmission method as channels for acoustic signals, and uses one or more channels of the plurality of channels of the acoustic transmission method different from the channels for the acoustic signals as channels for vibration signals, a conference signal generating unit for obtaining a signal of the acoustic transmission method, in which an acoustic signal input from the terminal device via the communication path is set as a signal for the channel for the acoustic signal, and no signal or an acoustic signal input from the terminal device via the communication path is set as a signal for the channel for the vibration signal, as a signal for a terminal device that presents sound but does not present vibration among the terminal devices. A multipoint conference control device comprising:
5. A multipoint conference control device included in a multipoint conference system including a plurality of terminal devices and a multipoint conference control device connected to each of the terminal devices via a communication path, the multipoint conference system uses one or more channels of a plurality of channels of a predetermined acoustic transmission method as channels for acoustic signals, and uses one or more channels of the plurality of channels of the acoustic transmission method different from the channels for the acoustic signals as channels for vibration signals, a conference signal generating unit for obtaining a signal of the acoustic transmission method, the conference signal generating unit generating a signal for a terminal device among the terminal devices that presents vibration but does not present sound, the signal being a vibration signal input from the terminal device through the communication path as a signal for the channel for the vibration signal, and a no signal or a vibration signal input from the terminal device through the communication path as a signal for the channel for the acoustic signal. A multipoint conference control device comprising:
6. A multipoint conference control device included in a multipoint conference system including a plurality of terminal devices and a multipoint conference control device connected to each of the terminal devices via a communication path, the multipoint conference system uses one or more channels of a plurality of channels of a predetermined acoustic transmission method as channels for acoustic signals, and uses one or more channels of the plurality of channels of the acoustic transmission method different from the channels for the acoustic signals as channels for vibration signals, a signal of the acoustic transmission method is obtained, as a signal for a terminal device among the terminal devices that presents sound but does not present vibration, in which an acoustic signal input from the terminal device via the communication path is set as a signal for the channel for the acoustic signal, and no signal or an acoustic signal input from the terminal device via the communication path is set as a signal for the channel for the vibration signal; a signal of the acoustic transmission method is obtained, as a signal for a terminal device among the terminal devices that presents vibration but does not present sound, in which a vibration signal input from the terminal device via the communication path is set as a signal on a channel for the vibration signal, and no signal or a vibration signal input from the terminal device via the communication path is set as a signal on a channel for the acoustic signal; a conference signal generating unit for obtaining signals of the acoustic transmission method, the conference signal generating unit generating signals for a terminal device that presents both sound and vibration among the terminal devices, the audio signal input from the terminal device through the communication path being a signal for the channel for the audio signal, and the vibration signal input from the terminal device through the communication path being a signal for the channel for the vibration signal. A multipoint conference control device comprising:
7. In a multipoint conference system including a plurality of terminal devices and a multipoint conference control device connected to each of the terminal devices via a communication path, a signal conversion device is disposed between the terminal devices and the multipoint conference control device, the multipoint conference system uses one or more channels of a plurality of channels of a predetermined acoustic transmission method as channels for acoustic signals, and uses one or more channels of the plurality of channels of the acoustic transmission method different from the channels for the acoustic signals as channels for vibration signals, the multipoint conference control device receives and outputs a signal of the acoustic transmission method in which an acoustic signal input from the terminal device via the communication path is treated as a signal of a channel for the acoustic signal, and a vibration signal input from the terminal device via the communication path is treated as a signal of a channel for the vibration signal, a conference signal conversion unit for obtaining a signal of the acoustic transmission method, the conference signal conversion unit converting an acoustic signal included in the signal of the acoustic transmission method output by the multipoint conference control device into a signal of a channel for the acoustic signal, and converting no signal or an acoustic signal included in the signal of the acoustic transmission method output by the multipoint conference control device into a signal of a channel for the vibration signal, as a signal for a terminal device among the terminal devices that can present sound but cannot present vibration. A signal conversion device comprising:
8. In a multipoint conference system including a plurality of terminal devices and a multipoint conference control device connected to each of the terminal devices via a communication path, a signal conversion device is disposed between the terminal devices and the multipoint conference control device, the multipoint conference system uses one or more channels of a plurality of channels of a predetermined acoustic transmission method as channels for acoustic signals, and uses one or more channels of the plurality of channels of the acoustic transmission method different from the channels for the acoustic signals as channels for vibration signals, the multipoint conference control device receives and outputs a signal of the acoustic transmission method in which an acoustic signal input from the terminal device via the communication path is treated as a signal of a channel for the acoustic signal, and a vibration signal input from the terminal device via the communication path is treated as a signal of a channel for the vibration signal, a conference signal conversion unit for obtaining a signal of the acoustic transmission method, the conference signal conversion unit converting a vibration signal included in the signal of the acoustic transmission method output by the multipoint conference control device into a signal of a channel for the vibration signal, and converting no signal or a vibration signal included in the signal of the acoustic transmission method output by the multipoint conference control device into a signal of the channel for the acoustic signal, as a signal for a terminal device among the terminal devices that can present vibration but cannot present sound. A signal conversion device comprising:
9. In a multipoint conference system including a plurality of terminal devices and a multipoint conference control device connected to each of the terminal devices via a communication path, a signal conversion device is disposed between the terminal devices and the multipoint conference control device, the multipoint conference control device receives and outputs signals of an acoustic transmission system in which an acoustic signal input from the terminal device via the communication path is treated as a signal on a channel for the acoustic signal, and a vibration signal input from the terminal device via the communication path is treated as a signal on a channel for the vibration signal, the multipoint conference system uses one or more of the multiple channels of the acoustic transmission method as channels for acoustic signals, and uses one or more of the multiple channels of the acoustic transmission method different from the channels for the acoustic signals as channels for vibration signals, obtaining a signal of the acoustic transmission method as a signal for a terminal device among the terminal devices that can present sound but cannot present vibration, the signal being an acoustic signal included in the signal of the acoustic transmission method output by the multipoint conference control device as a signal on a channel for the acoustic signal, and no signal or an acoustic signal included in the signal of the acoustic transmission method output by the multipoint conference control device as a signal on a channel for the vibration signal; obtaining a signal of the acoustic transmission method, as a signal for a terminal device among the terminal devices that can present vibration but cannot present sound, in which a vibration signal included in the signal of the acoustic transmission method output by the multipoint conference control device is set as a signal on a channel for the vibration signal, and no signal or a vibration signal included in the signal of the acoustic transmission method output by the multipoint conference control device is set as a signal on a channel for the acoustic signal; a conference signal conversion unit for obtaining a signal of the acoustic transmission method, the conference signal conversion unit converting an acoustic signal included in the signal of the acoustic transmission method output by the multipoint conference control device into a signal of a channel for the acoustic signal and a vibration signal included in the signal of the acoustic transmission method output by the multipoint conference control device into a signal of a channel for the vibration signal as a signal for a terminal device that can present both sound and vibration among the terminal devices; A signal conversion device comprising:
10. In a multipoint conference system including a plurality of terminal devices and a multipoint conference control device connected to each of the terminal devices via a communication path, a signal conversion device is disposed between the terminal devices and the multipoint conference control device, the multipoint conference system uses one or more channels of a plurality of channels of a predetermined acoustic transmission method as channels for acoustic signals, and uses one or more channels of the plurality of channels of the acoustic transmission method different from the channels for the acoustic signals as channels for vibration signals, the multipoint conference control device receives and outputs a signal of the acoustic transmission method in which an acoustic signal input from the terminal device via the communication path is treated as a signal of a channel for the acoustic signal, and a vibration signal input from the terminal device via the communication path is treated as a signal of a channel for the vibration signal, a conference signal conversion unit for obtaining a signal of the acoustic transmission method, the conference signal conversion unit converting an acoustic signal included in the signal of the acoustic transmission method output by the multipoint conference control device into a signal of a channel for the acoustic signal, and converting no signal or an acoustic signal included in the signal of the acoustic transmission method output by the multipoint conference control device into a signal of a channel for the vibration signal, as a signal for a terminal device among the terminal devices that presents sound but does not present vibration. A signal conversion device comprising:
11. In a multipoint conference system including a plurality of terminal devices and a multipoint conference control device connected to each of the terminal devices via a communication path, a signal conversion device is disposed between the terminal devices and the multipoint conference control device, the multipoint conference system uses one or more channels of a plurality of channels of a predetermined acoustic transmission method as channels for acoustic signals, and uses one or more channels of the plurality of channels of the acoustic transmission method different from the channels for the acoustic signals as channels for vibration signals, the multipoint conference control device receives and outputs a signal of the acoustic transmission method in which an acoustic signal input from the terminal device via the communication path is treated as a signal of a channel for the acoustic signal, and a vibration signal input from the terminal device via the communication path is treated as a signal of a channel for the vibration signal, a conference signal conversion unit for obtaining a signal of the acoustic transmission method, the conference signal conversion unit converting a vibration signal included in the signal of the acoustic transmission method output by the multipoint conference control device into a signal of a channel for the vibration signal, and converting no signal or a vibration signal included in the signal of the acoustic transmission method output by the multipoint conference control device into a signal of the channel for the acoustic signal, as a signal for a terminal device among the terminal devices that presents vibration but does not present sound. A signal conversion device comprising:
12. In a multipoint conference system including a plurality of terminal devices and a multipoint conference control device connected to each of the terminal devices via a communication path, a signal conversion device is disposed between the terminal devices and the multipoint conference control device, the multipoint conference control device receives and outputs signals of an acoustic transmission system in which an acoustic signal input from the terminal device via the communication path is treated as a signal on a channel for the acoustic signal, and a vibration signal input from the terminal device via the communication path is treated as a signal on a channel for the vibration signal, the multipoint conference system uses one or more of the multiple channels of the acoustic transmission method as channels for acoustic signals, and uses one or more of the multiple channels of the acoustic transmission method different from the channels for the acoustic signals as channels for vibration signals, obtaining a signal of the acoustic transmission method, as a signal for a terminal device among the terminal devices that presents sound but not vibration, in which an acoustic signal included in the signal of the acoustic transmission method output by the multipoint conference control device is set as a signal on a channel for the acoustic signal, and no signal or an acoustic signal included in the signal of the acoustic transmission method output by the multipoint conference control device is set as a signal on a channel for the vibration signal; obtaining a signal of the acoustic transmission method, as a signal for a terminal device among the terminal devices that presents vibration but does not present sound, in which a vibration signal included in the signal of the acoustic transmission method output by the multipoint conference control device is set as a signal on a channel for the vibration signal, and no signal or a vibration signal included in the signal of the acoustic transmission method output by the multipoint conference control device is set as a signal on a channel for the acoustic signal; As a signal for a terminal device that presents both sound and vibration among the terminal devices, a signal of the acoustic transmission method is obtained in which an acoustic signal included in the signal of the acoustic transmission method output by the multipoint conference control device is set as a signal of a channel for the acoustic signal, and a vibration signal included in the signal of the acoustic transmission method output by the multipoint conference control device is set as a signal of a channel for the vibration signal. Conference signal converter A signal conversion device comprising:
13. A terminal device included in a multipoint conference system including a plurality of terminal devices and a multipoint conference control device connected to each of the terminal devices via a communication path, the multipoint conference system uses one or more channels of a plurality of channels of a predetermined acoustic transmission method as channels for acoustic signals, and uses one or more channels of the plurality of channels of the acoustic transmission method different from the channels for the acoustic signals as channels for vibration signals, Equipped with a display screen, On the display screen, a media group switching button which, when pressed, performs a selection by cyclic state switching including at least three states: a state in which sound is presented without vibration, a state in which both sound and vibration are presented, and a state in which vibration is presented without sound; a media display group that, when one of the states is selected by pressing the media group switching button, displays in a visually recognizable manner whether the state is a state in which sound is presented without vibration, a state in which both sound and vibration are presented, or a state in which vibration is presented without sound; a presentation signal selection user interface region including A terminal device comprising:
14. The terminal device according to claim 13, The media display group includes: The color and / or brightness of a button (hereinafter referred to as the "sound button") indicates whether or not a sound is to be presented, The button (hereinafter, referred to as the "vibration button") indicates whether or not the device is in a state in which vibration is to be presented by changing the color and / or brightness of the button. The terminal device When the sound button is pressed, a switch is made as to whether or not to present a sound; When the vibration button is pressed, it is switched whether or not vibration is presented. A terminal device comprising:
15. A program for causing a computer to function as the multipoint conference control device according to any one of claims 1 to 6.
16. A program for causing a computer to function as the signal conversion device according to any one of claims 7 to 12.
17. A program for causing a computer to function as the terminal device according to claim 13 or 14.
Citation Information
Patent Citations
Environmental information acquiring and reproducing system, video conference system, vehicle, clothing, and remote control system
JP2006191460A
Terminal control device and terminal control method using the same
JP2018502486A
Device, method, and program for delivering haptic vibration
JP2021047756A
Acoustic vibration reproduction system and acoustic vibration reproduction method
JP2022024649A
Cognitive enhancement of communication with tactile stimulation
US20200168205A1
