Information processing device, transmission system and program

Abstract

To provide an information processing device which reports other information processing device recording image data or voice data.SOLUTION: An information processing device 10, which transmits image data or voice data to one or more other information processing devices 40 connected through a network, includes: reception means 11 which receives record notification, which indicates that the image data or the voice data transmitted from the information processing device is recorded by the other the information processing devices, together with the identification information of the other information processing devices; and record terminal notification means 18b which, based on the identification information received by the reception means, notifies that the other information processing devices which transmit the record notification is recording the image data or the voice data in a manner to associate with the other information processing devices.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a communication system, and a program. [Background technology]

[0002] There is known a transmission system for conducting a video conference or the like between a plurality of terminal devices via a network such as the Internet. In such a transmission system, one transmission terminal transmits image data and audio data collected during a video conference to another transmission terminal. The other transmission terminal receives the image data and audio data and displays the image on a display or outputs the audio from a speaker, thereby enabling a video conference between these transmission terminals.

[0003] In such a transmission system, the number of transmission terminals participating in a conference is not limited to two, and it is not rare that a video conference is held between multiple locations with three or more transmission terminals participating in the video conference.

[0004] Furthermore, while each location can display video and output audio as described above, the transmission terminal at each location may also have the function of recording video and audio (see, for example, Patent Document 1). This is to allow those who were unable to participate in the video conference to understand or confirm the contents of the conference later. Summary of the Invention [Problem to be solved by the invention]

[0005] However, since images and audio transmitted in video conferences often contain highly confidential information, many participants do not want to be recorded or videotaped. Furthermore, some participants do not want their portraits included in the images to be recorded. In this regard, the communication system disclosed in Patent Document 1 uses a technology in which each location records communication information on the same recording device, which creates a problem in that if a location is recording or videotaping, it is difficult for participants at other locations to identify the location that is recording or videotaping.

[0006] For example, if a video conference is being held at locations A to C, and location C is recording images or audio, the participants at location A will not know whether location B or C is recording the video or audio, and the participants at location B will not know whether location A or C is recording the video or audio. This makes it difficult to determine which location should be asked to stop recording, and whether highly confidential information should be disclosed.

[0007] SUMMARY OF THE INVENTION In view of the above-mentioned problems, an object of the present invention is to provide an information processing device that notifies other information processing devices that are recording image data or audio data. [Means for solving the problem]

[0008] The present invention is an information processing device that transmits at least one of image data or audio data to one or more other information processing devices connected via a network, and includes a receiving means that receives a recording notification indicating that the other information processing device will record the image data or audio data transmitted by the information processing device, along with identification information of the other information processing device, and a recording terminal notification means that, based on the identification information received by the receiving means, notifies the other information processing device that the other information processing device that transmitted the recording notification is currently recording image data or audio data, in association with the other information processing device. [Effects of the Invention]

[0009] It is possible to provide an information processing device that notifies other information processing devices that are recording image data or audio data. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of an outline of the operation of a transmission system. [Figure 2] 1 is a schematic diagram of an example of a transmission system; [Figure 3] 1 is a conceptual diagram showing the transmission and reception states of image data, audio data, and various management information in a transmission system. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a transmission terminal. [Figure 5] FIG. 2 is a hardware configuration diagram of a transmission management device, a UI providing device, a relay device, or a program providing device. [Figure 6] FIG. 2 is an example of a functional block diagram of each terminal included in the transmission system. [Figure 7] FIG. 2 is a functional block diagram of an information processing terminal; [Figure 8] An example is shown in which a transmitting terminal goes into a ready state. [Figure 9] 10 is a sequence diagram illustrating an example of a conference participation operation until a transmission terminal is in a participating state to participate in a video conference. [Figure 10] 10 is a sequence diagram illustrating an example of an operation when the information processing terminal starts and ends recording. [Figure 11] FIG. 9 is a diagram showing an example of a destination list screen displayed in step S32 of FIG. 8. [Figure 12] 10 is a diagram illustrating an example of a confirmation dialog screen displayed when a user selects a destination of a recording location mark. [Figure 13] FIG. 10 is a diagram showing an example of a screen displayed on a display in full screen display mode. [Figure 14] FIG. 10 is a diagram showing an example of an auxiliary area in a simplified display. [Figure 15] 10A and 10B are diagrams showing examples of how other location recording marks and labels are displayed when the auxiliary area is not displayed. [Figure 16] This is an example of a full-screen display screen when there is no participating location that is recording. [Figure 17] 10 is a flowchart illustrating an example of a procedure in which a transmission terminal during a conference controls display of recording notification information. [Figure 18] 10 is a sequence diagram illustrating an example of an operation procedure of the transmission system 1 when the transmission management device detects that a session has been disconnected. [Figure 19] 10 is a flowchart illustrating an example of processing performed when a transmission terminal receives terminal state change information. [Figure 20] FIG. 10 is a diagram showing an example of an information display screen displayed on a display in an information display mode. [Figure 21] 10 is a sequence diagram illustrating an example of a terminal information display operation for displaying terminal information in a terminal information display area when the transmission terminal is switched to an information display mode. [Figure 22] FIG. 1 is a diagram illustrating an example of an electronic whiteboard having a function as a transmission terminal. [Figure 23] 1 is a diagram illustrating an example of an outline of the operation of an electronic whiteboard; [Figure 24] 10 is a diagram showing an example of an electronic whiteboard display on which an electronic whiteboard UI screen generated by an electronic whiteboard application is displayed. [Figure 25] FIG. 10 is a diagram illustrating an example of an interactive whiteboard display on which a video conference UI screen generated by a video conference application is displayed. [Figure 26] FIG. 2 is a hardware configuration diagram of an electronic whiteboard. [Figure 27] FIG. 2 is an example of a functional block diagram of an electronic whiteboard. [Figure 28] FIG. 2 is a diagram illustrating an example of the configuration of each image layer. [Figure 29] FIG. 2 is a diagram conceptually illustrating page data. [Figure 30] FIG. 2 is a diagram conceptually illustrating stroke arrangement data. [Figure 31] FIG. 2 is a diagram conceptually showing coordinate array data. [Figure 32] FIG. 2 is a diagram conceptually illustrating media data. [Figure 33] 10 is a sequence diagram illustrating an example of an operation procedure when a video conferencing application running on an electronic whiteboard receives a recording start notification. [Figure 34] 10 is an example of an interactive whiteboard UI screen that displays information indicating that another location is recording. [Figure 35] 10 is a sequence diagram illustrating an example of a procedure in which an interface unit changes the display order of an electronic whiteboard UI screen and a video conference UI screen. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. [Example]

[0012] 1 is an example of a diagram illustrating an outline of the operation of the transmission system 1 according to the present embodiment. The transmission terminal 10aa is holding a video conference with the transmission terminal 10ba and the information processing terminal 40bd. The transmission terminal 10aa combines images from the transmission terminal 10ba and the information processing terminal 40bd and displays the combined image on the display 120 (such a screen is referred to as a full-screen display screen 501). The transmission terminal 10ba and the information processing terminal 40bd also display the same image.

[0013] (1) In this embodiment, when the information processing terminal 40bd starts recording an image during a video conference, it transmits a recording start notification (an example of a recording notification) to the transmitting terminals 10aa and 10ba. For convenience of explanation, the following description focuses on the transmitting terminal 10aa.

[0014] (2) The transmission terminal 10aa holds the communication ID (04bd) of the information processing terminal 40bd included in the recording start notification.

[0015] (3) When the user of the transmission terminal 10aa displays the information display screen 601 to check the location where the recording is being performed, the location where the recording is being performed is highlighted using the stored communication ID. In FIG. 1, the terminal name and communication ID are displayed, and a recording location mark 71 is displayed at the beginning of the terminal name of the information processing terminal 40bd that is performing the recording. The user of the transmission terminal 10aa can determine which of the multiple locations the recording is being performed at by using the recording location mark 71.

[0016] In this way, in the transmission system 1 of this embodiment, when a conference is being held at another location, participants at the other location can understand, confirm, or identify in real time which location is recording.

[0017] <Terminology> The identification information of an information processing device is information that allows other information processing devices to identify the information processing device. Specifically, a communication ID can be used. In addition, the information processing device may be identified by a terminal name, a MAC address, an email address, a user name, a user ID, or the like.

[0018] The information processing device is a device capable of transmitting and receiving at least one of image data and audio data. In this embodiment, the information processing device will be described using the terms transmission terminal 10 and information processing terminal 40.

[0019] Notifying another information processing device that recording is in progress in association with the other information processing device means outputting information that can identify the information processing device, such as the terminal name or communication ID of the information processing device, along with information indicating that at least one of image data and audio data is being recorded. Output includes visual as well as audio notification. In this embodiment, for example, a recording location mark 71 is displayed in association with the terminal name and communication ID.

[0020] Furthermore, recording of image data or audio data refers to a process that can be expressed as preservation, retention, storage, or memorization. Recorded image data or audio data can be played back. Recording can also be referred to as video recording or audio recording.

[0021] The information about other information processing devices refers to information that enables a participant to know other participants in the conference, such as at least one of the terminal name and the communication ID.

[0022] The term "in communication" refers to a state in which two or more transmission terminals 10 and information processing terminals 40 are transmitting and receiving image data or audio data. Specifically, the term "in communication" refers to a state in which a video conference is being held or a state in which a video conference is being participated in.

[0023] A recording notification is a notification that recording has started or is currently being recorded, and a recording end notification is a notification that recording has ended.

[0024] <Overall configuration of the embodiment> FIG. 2 is a schematic diagram of a transmission system 1 according to this embodiment. FIG. 3 is a conceptual diagram showing the transmission and reception states of image data, audio data, and various management information in the transmission system 1. The transmission system 1 includes a data providing system that transmits content data unidirectionally from one transmission terminal to multiple transmission terminals via a transmission management device 50. It also includes a communication system that transmits information, emotions, etc. between multiple transmission terminals via the transmission management device 50. The communication system is a system for mutually transmitting information, emotions, etc. between multiple communication terminals (corresponding to "transmission terminals") via a communication management device (corresponding to "transmission management device 50"). Examples include a video conference system and a video telephone system.

[0025] In this embodiment, the transmission system 1, the transmission management device 50, and the transmission terminal 10 will be described assuming a video conference system as an example of a communication system, a video conference management device as an example of a communication management device, and a video conference terminal as an example of a communication terminal. That is, the transmission terminal and the transmission management device 50 of the present invention are not only applied to a video conference system, but also to a communication system or a data transmission system.

[0026] 2, the transmission system 1 includes a plurality of transmission terminals 10aa, 10ab, ..., displays 120aa, 120ab, ... for each of the transmission terminals 10aa, 10ab, ..., a UI providing device 20, information processing terminals 40bd, 40be, a relay device 30, a transmission management device 50, and a program providing device 90. The transmission terminals 10aa, 10ab, ... are configured to transmit content data, for example, by sending and receiving image data and audio data.

[0027] The UI providing device 20 is a device that transmits layout information (to be described later) for mainly displaying the number of locations to the transmission terminals 10aa, 10ab, etc. The UI providing device 20 may also be implemented by the transmission management device 50 or the relay device 30, for example.

[0028] The information processing terminal 40 is a device in which the functions of the transmission terminal 10 are realized by software. Specifically, the information processing terminal 40 is, for example, a personal computer (PC), a tablet terminal, a smartphone, a mobile phone, a personal digital assistant (PDA), a game console, a car navigation device, or an electronic book terminal.

[0029] In the following description, any one of the multiple transmission terminals 10aa, 10ab, ... will be referred to as a "transmission terminal 10," and any one of the multiple displays 120aa, 120ab, ... will be referred to as a "display 120." Any one of the multiple information processing terminals 40bd, 40be will be referred to as an "information processing terminal 40." Furthermore, the transmission terminal 10 that requests the start of a videoconference will be referred to as a "requesting terminal," and the transmission terminal 10 that is the request recipient will be referred to as a "destination terminal."

[0030] Furthermore, a transmitting terminal 10 participating in a video conference or a location where the transmitting terminal 10 is assumed to be installed is referred to as a "base." Alternatively, it may be referred to as a partner user using the transmitting terminal 10. Once the transmitting terminal 10 participating in the video conference is identified, the user can estimate the location where the transmitting terminal 10 is assumed to be installed and the partner user. Using the terminal name described below, the user can estimate the location where the partner or the transmitting terminal 10 is assumed to be installed. Furthermore, it is not necessary to identify the location of the partner; the location where the partner is located is the base. Furthermore, it is not necessary to identify the partner as an individual; it is sufficient to know attributes such as the company name. For example, in corporate use, once the partner's transmission terminal 10 is identified, the user can estimate the partner and their location. The number of bases is the same as the number of transmitting terminals 10 participating in the video conference; however, multiple transmitting terminals 10 may exist in a space that can be considered as one (for example, one conference room).

[0031] As shown in Figure 3, in the transmission system 1, a management information session sei is established between the requesting terminal, that is, the transmission terminal 10aa, and the destination terminal, that is, the transmission terminal 10ab (the symbol is an example), via the transmission management device 50, for transmitting and receiving various management information.

[0032] Furthermore, between the transmission terminals 10 where the management information session sei has been established, four sessions for transmitting and receiving four types of data, namely, high-resolution image data sed1, medium-resolution image data sed2, low-resolution image data sed3, and audio data sed4, are established via the relay device 30. In this embodiment, these four sessions are collectively referred to as the "image and audio data session sed."

[0033] 2, the relay device 30 relays content data among a plurality of transmission terminals 10. The transmission management device 50 centrally manages login authentication from the transmission terminals 10, management of the transmission status of the transmission terminals 10, management of the destination list, and the transmission status of the relay device 30. Note that the image data may represent a moving image or a still image, or may represent both a moving image and a still image.

[0034] The multiple routers 70a, 70b, 70c, 70d, 70ab, and 70cd select the optimal route for image data and audio data. In the following description, any one of the routers 70a, 70b, 70c, 70d, 70ab, and 70cd will be referred to as a "router 70."

[0035] The program providing device 90 is equipped with an HD (Hard Disk) 204, which will be described later, and stores a transmission terminal control program 1000a for realizing various functions in the transmission terminal 10, and can transmit the transmission terminal control program 1000a to the transmission terminal 10.

[0036] Furthermore, the HD 204 of the program providing device 90 also stores a relay device program for causing the relay device 30 to realize various functions, and can transmit the relay device program to the relay device 30.

[0037] Furthermore, the HD 204 of the program providing device 90 also stores a transmission management program 5000 a for causing the transmission management device 50 to realize various functions, and the transmission management program 5000 a can be transmitted to the transmission management device 50 .

[0038] Furthermore, the HD 204 of the program providing device 90 also stores a program for the UI providing device for causing the UI providing device 20 to realize various functions, and can transmit the program for the UI providing device to the UI providing device 20 .

[0039] In addition, the HD 204 of the program providing device 90 also stores a program for information processing terminal 1000c for realizing various functions in the information processing terminal 40, and can transmit the program for information processing terminal 1000c to the information processing terminal 40.

[0040] The transmission terminals 10aa, 10ab, 10ac, and so on and the router 70a are connected to each other via a LAN 2a so as to be able to communicate with each other. The transmission terminals 10ba, 10bb, 10bc, and so on and the router 70b are connected to each other via a LAN 2b so as to be able to communicate with each other.

[0041] Furthermore, LAN2a and LAN2b are connected to each other so that communication is possible via a dedicated line 2ab including a router 70ab, and are provided within a predetermined area A. For example, area A is Japan, with LAN2a provided within an office in Tokyo and LAN2b provided within an office in Osaka.

[0042] On the other hand, the transmission terminals 10ca, 10cb, 10cc, etc. and the router 70c are connected to each other via a LAN 2c so as to be able to communicate with each other. The transmission terminals 10da, 10db, 10dc, etc. and the router 70d are connected to each other via a LAN 2d so as to be able to communicate with each other.

[0043] Furthermore, LAN2c and LAN2d are connected to each other so that they can communicate with each other via a dedicated line 2cd that includes a router 70cd, and are located within a predetermined area B. For example, area B is the United States of America, with LAN2c located within an office in New York and LAN2d located within an office in Washington, D.C. Areas A and B are connected to each other so that they can communicate with each other via routers 70ab and 70cd, respectively, via the Internet 2i.

[0044] Furthermore, the relay device 30, the transmission management device 50, the UI providing device 20, the information processing terminal 40, and the program providing device 90 are connected to each transmission terminal 10 via the Internet 2i so as to be able to communicate with each other. Here, the relay device 30, the transmission management device 50, the UI providing device 20, the information processing terminal 40, and the program providing device 90 may be installed in area A or area B, or may be installed in an area other than these.

[0045] In this embodiment, the transmission network 2 of this embodiment is configured by the LAN 2a, the LAN 2b, the dedicated line 2ab, the Internet 2i, the dedicated line 2cd, the LAN 2c, and the LAN 2d. The transmission network 2 may include locations where transmission is not only performed by wire but also by wireless such as WiFi (Wireless Fidelity) or Bluetooth (registered trademark).

[0046] 2, four sets of numbers shown under each transmission terminal 10, relay device 30, transmission management device 50, each router 70, UI providing device 20, information processing terminal 40, and program providing device 90 simply indicate IP addresses in general IPv4. For example, the IP address of the transmission terminal 10aa is "1.2.1.3." Although IPv6 may be used instead of IPv4, for the sake of simplicity, IPv4 will be used in the following description.

[0047] Each transmission terminal 10 may be used not only for video conferences between multiple offices or between different rooms in the same office, but also for video conferences in the same room, or for video conferences between indoors and outdoors or between outdoors and outdoors. When each transmission terminal 10 is used outdoors, wireless transmission is performed via a mobile phone transmission network or the like.

[0048] <Hardware configuration of transmission terminal> 4 is an example of a hardware configuration diagram of a transmission terminal. As shown in FIG. 4, the transmission terminal 10 includes a CPU (Central Processing Unit) 101 that controls the overall operation of the transmission terminal 10, and a ROM (Read Only Memory) 102 that stores programs used to drive the CPU 101, such as an IPL (Initial Program Loader). The transmission terminal 10 also includes a RAM (Random Access Memory) 103 used as a work area for the CPU 101, and a flash memory 104 that stores a transmission terminal control program 1000a and various data such as image data and audio data. The transmission terminal 10 also includes a SSD (Solid State Drive) 105 that controls reading and writing of various data from and to the flash memory 104 under the control of the CPU 101. The transmission terminal 10 also includes a media drive 107 that controls reading and writing (storing) of data from and to a recording medium 106 such as a flash memory, and operation buttons 108 that are operated, for example, when selecting a destination for the transmission terminal 10. The transmission terminal 10 also includes a power switch 109 for switching the power of the transmission terminal 10 on and off, and a network I / F (Interface) 111 for transmitting data using the transmission network 2.

[0049] The transmission terminal 10 also has a built-in camera 112 that captures an image of a subject and obtains image data under the control of the CPU 101, an image sensor I / F 113 that controls the driving of the camera 112, and a microphone 114 as a built-in sound collector that inputs sound.The transmission terminal 10 also has a built-in speaker 115 that outputs sound, and an audio input / output I / F 116 that processes input and output of audio signals between the microphone 114 and the speaker 115 under the control of the CPU 101.The transmission terminal 10 also has a display I / F 117 that transmits image data to an external display 120 under the control of the CPU 101.The transmission terminal 10 also has an external device connection I / F 118 for connecting various external devices, and bus lines 110 such as an address bus and a data bus for electrically connecting the above components.

[0050] The display 120 is a display device configured with a liquid crystal or organic EL display that displays an image of a subject, operation icons, etc. The display 120 is connected to the display I / F 117 by a cable 120c. The cable 120c may be a cable for analog RGB (VGA) signals, a cable for component video, or a cable for HDMI (registered trademark) (High-Definition Multimedia Interface) or DVI (Digital Video Interactive) signals.

[0051] The camera 112 includes a lens and a solid-state image sensor that converts light into electric charges to digitize an image of a subject. The solid-state image sensor may be a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD).

[0052] The external device connection I / F 118 allows external devices such as an external camera, an external microphone, and an external speaker to be connected via a USB (Universal Serial Bus) cable or the like.

[0053] When an external camera is connected to the external device connection I / F 118, the CPU 101 drives the external camera in priority over the built-in camera 112. When an external microphone or an external speaker is connected to the external device connection I / F 118, the CPU 101 drives the external microphone or the external speaker in priority over the built-in microphone 114 or the built-in speaker 115, respectively.

[0054] The recording medium 106 is configured to be detachable from the transmission terminal 10. The transmission terminal 10 may use an EEPROM (Electrically Erasable and Programmable ROM) or the like instead of the flash memory 104, as long as the EEPROM is a nonvolatile memory that reads or writes data under the control of the CPU 101.

[0055] The transmission terminal control program 1000a may be distributed by being recorded in an installable or executable file format on a computer-readable recording medium such as the recording medium 106. The transmission terminal control program 1000a may be stored in the ROM 102 instead of the flash memory 104.

[0056] The information processing terminal 40 has the same hardware configuration as the transmission terminal 10, or even if there are differences, they will not hinder the description of this embodiment. However, the SSD 105 stores an information processing terminal program 1000c for controlling the information processing terminal 40.

[0057] <<Hardware configuration diagram of the transmission management device 50, UI providing device, relay device, and program providing device>> FIG. 5 is a hardware configuration diagram of the transmission management device 50, UI providing device 20, relay device, or program providing device. As shown in FIG. 5, the transmission management device 50 includes a CPU 201 that controls the overall operation of the transmission management device 50, a ROM 202 that stores programs used to drive the CPU 201, such as an IPL, and a RAM 203 that serves as a work area for the CPU 201. The transmission management device 50 also includes a hard disk drive (HDD) 205 that stores various data, such as the transmission management program, and controls the reading and writing of various data from the HDD 204 under the control of the CPU 201. The transmission management device 50 also includes a media drive 207 that controls the reading and writing (storage) of data from a recording medium 206, such as a flash memory, and a display 208 that displays various information, such as a cursor, menu, window, text, or image. The transmission management device 50 also includes a network interface 2809 for transmitting data using the transmission network 2 and a keyboard 211 with multiple keys for inputting text, numbers, various instructions, and the like. The computer also has a mouse 212 for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. It also has an optical drive 214 for controlling reading and writing of various data from and to an optical storage medium (Compact Disc Read Only Memory) 213, which is an example of a removable storage medium. It also has bus lines 210 such as an address bus and a data bus for electrically connecting the above components.

[0058] The transmission management program may be distributed by being recorded in an installable or executable file format on a computer-readable recording medium such as the recording medium 206 or the optical storage medium 213. The transmission management program may also be stored in the ROM 202 instead of the HD 204.

[0059] Furthermore, the relay device 30 has the same hardware configuration as the transmission management device 50, and therefore a description thereof will be omitted. However, the HD 204 stores a relay device program for controlling the relay device 30.

[0060] In this case, the program for the relay device may also be distributed by being recorded in an installable or executable file format on a computer-readable recording medium such as the recording medium 206 or the optical storage medium 213. The program for the relay device may also be stored in the ROM 202 instead of the HD 204.

[0061] Furthermore, the program providing device 90 has the same hardware configuration as the transmission management device 50, and therefore its description will be omitted. However, the HD 204 stores a program for the program providing device for controlling the program providing device 90.

[0062] In this case, too, the program for the program providing device may be distributed by being recorded as an installable or executable file on a computer-readable recording medium such as the recording medium 206 or the optical storage medium 213. The program for the program providing device may also be stored in the ROM 202 instead of the HD 204.

[0063] The UI providing device 20 has the same hardware configuration as the transmission management device 50, and therefore its description will be omitted. However, the HD 204 stores a UI providing device program for controlling the UI providing device 20 and layout information, which will be described later.

[0064] In this case, too, the UI providing device program and layout information may be distributed by being recorded in an installable or executable file format on a computer-readable recording medium such as the recording medium 206 or the optical storage medium 213. The UI providing device program and layout information may also be stored in the ROM 202 instead of the HD 204.

[0065] The computer-readable recording medium may be a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), a Blu-ray disc, or the like.

[0066] It should be noted that the illustrated hardware configuration of the transmission management device 50 etc. does not need to be housed in a single housing or provided as a single integrated device, but indicates hardware elements that are preferably provided in the transmission management device 50 etc. Also, in order to accommodate cloud computing, the physical configuration of the transmission management system 50 etc. of this embodiment does not need to be fixed, but may be configured such that hardware resources are dynamically connected and disconnected according to the load.

[0067] <Functional configuration of transmission system 1> Fig. 6 is a functional block diagram of each terminal included in the transmission system 1. The transmission terminal 10, the relay device 30, the information processing terminal 40, the UI providing device 20, and the transmission management device 50 are connected so as to be able to transmit data via the transmission network 2. Note that the relay device 30 and the program providing device 90 shown in Fig. 2 will be explained as needed. The information processing terminal 40 will be explained in Fig. 7.

[0068] <<Functional configuration of the transmission terminal>> The transmission terminal 10 has a transmission / reception unit 11, an operation input reception unit 12, a login request unit 13, an imaging unit 14, a sound input unit 15a, a sound output unit 15b, a destination list display unit 16, a display control unit 17, and a storage / readout processing unit 19. These units are functions realized by a CPU 101 that executes a transmission terminal control program 1000a that is loaded from the flash memory 104 to the RAM 103 in FIG. 4. The transmission terminal 10 also has a storage unit 1000 that is configured by the RAM 103 and the flash memory 104. The storage unit 1000 stores the transmission terminal control program 1000a and a recording in progress management table 1000b.

[0069] [Table 1] Table 1 shows an example of the recording in progress management table 1000b. The communication IDs of the transmitting terminal 10 and the information processing terminal 40 that are recording are registered in the recording in progress management table 1000b. That is, the transmitting terminal 10 and the information processing terminal 40 that have started recording transmit a recording start notification (including their communication IDs) to the other transmitting terminals 10 and the information processing terminals 40, and the transmitting terminals 10 and the information processing terminals 40 that have received the recording start notification register their communication IDs in the recording in progress management table 1000b. When multiple transmitting terminals 10 and information processing terminals 40 are recording, the recording start notifications transmitted by the communication IDs of all the transmitting terminals 10 and the information processing terminals 40 that transmitted the recording start notifications are held in the recording in progress management table 1000b.

[0070] For convenience of explanation, it is assumed that a communication ID starting with "01" means the communication ID of the transmission terminal 10, and a communication ID starting with "04" means the communication ID of the information processing terminal 40.

[0071] (Functional configuration of the transmission terminal) The transmission / reception unit 11 is composed of a CPU 101, a network I / F 111, etc., and is configured to transmit and receive various data and information with other transmission terminals 10, information processing terminals 40, relay devices 30, transmission management devices 50, or program providing devices 90 via the transmission network 2.

[0072] Before starting a video conference with a desired destination terminal, the transmitter / receiver 11 starts receiving status information indicating the status of each transmitting terminal 10 as a candidate for connection to the video conference from the transmission management device 50. The status information not only indicates the operating status of each transmitting terminal 10, such as whether it is online or offline, but also indicates detailed operating status such as whether it is online and ready for transmission, whether it is currently transmitting online, etc.

[0073] Furthermore, the status information indicates not only the operating status of each transmission terminal 10 but also various other statuses, such as the cable 120c being disconnected from the transmission terminal 10, outputting sound but not outputting images, not outputting sound (MUTE), etc. In the following description, to facilitate understanding of the present invention, a case will be described in which the status information indicates the operating status of each transmission terminal 10.

[0074] The operation input receiving unit 12 is configured with a CPU 101, an operation button 108, a power switch 109, etc., and is configured to receive various inputs from a user. For example, when the user turns on the power switch 109, the operation input receiving unit 12 receives the power-on command and turns on the power of the transmission terminal 10.

[0075] The login request unit 13 is configured by the CPU 101 etc. When the power is turned on, the transmitter / receiver 11 transmits login request information indicating a login request and the current IP address of the terminal to the transmission management device 50 via the transmission network 2.

[0076] Furthermore, when the user turns the power switch 109 from an ON state to an OFF state, the transmitting / receiving unit 11 transmits state information indicating that the power is to be turned OFF to the transmission management device 50, and then the operation input receiving unit 12 completely turns OFF the power of the transmission terminal 10. This allows the transmission management device 50 to know that the transmission terminal 10 has been turned OFF from a power ON state.

[0077] The imaging unit 14 is configured with a CPU 101, a camera 112, an imaging element I / F 113, etc., and is configured to capture an image of a subject and output image data obtained by capturing the image to the transmitting / receiving unit 11.

[0078] The sound input unit 15a is composed of a CPU 101, a sound input / output I / F 116, etc., and receives a sound signal representing a sound collected by a microphone 114 and converts the input sound signal into audio data.

[0079] The sound output unit 15b is composed of the CPU 101, the sound input / output I / F 116, etc., and converts audio data into a sound signal and outputs the converted sound signal to the speaker 115, thereby causing the speaker 115 to output sound.

[0080] The display control unit 17 is configured with the CPU 101, a display I / F 117, etc. The display control unit 17 combines image data with different resolutions received by the transmitting / receiving unit 11 and transmits the combined image data to the display 120, thereby displaying an image represented by the image data on the display 120 (full screen display mode). It can also display detailed information about other locations (information display mode).

[0081] The display control unit 17 has an auxiliary area display unit 18a and a base list display unit 18b. When the transmitting terminal 10 is in a full screen display mode (to be described later), the auxiliary area display unit 18a displays the number of bases participating in the conference, recording notification information (recording notification information), and the like in the auxiliary area of ​​the full screen display screen 501. When the transmitting terminal 10 is in an information display mode (to be described later), the base list display unit 18b displays the names of the bases participating in the conference, their communication IDs, and recording base mark 71 and the like on the information display screen 601 by referring to the recording management table 1000b.

[0082] The destination list display unit 16 displays a destination list screen on the display 120 based on the destination status information transmitted from the transmission management device 50. The destination status information includes an operating status of whether or not a recording conference is in progress, and the destination list screen displays whether or not each of the destination transmission terminals 10 and information processing terminals 40 is in a conference.

[0083] The storage / readout processing unit 19 is configured by the CPU 101, the SSD 105, etc., and stores various data in the storage unit 1000 and reads out various data stored in the storage unit 1000.

[0084] The storage unit 1000 stores a communication ID, a password, and the like for identifying the transmitting terminal 10 as a communication destination (seen from another location). Furthermore, the storage unit 1000 is also used as a receiving buffer for image data and audio data received when a video conference is held with a destination terminal. Note that the communication ID in this embodiment indicates information such as language, characters, symbols, or various marks used to uniquely identify the transmitting terminal 10. The communication ID may also be information that combines at least two of the language, characters, symbols, and various marks.

[0085] <Functional configuration of the transmission management device 50> The transmission management device 50 has a transmitting / receiving unit 51, a terminal authentication unit 52, a status management unit 53, a terminal extraction unit 54, a terminal status acquisition unit 55, a session management unit 56, and a storage / readout processing unit 59. Each of these units is a function realized by a CPU 201 that executes a transmission management program 5000a expanded from the HD 204 onto the RAM 203 shown in Fig. 5. The transmission management device 50 also has a storage unit 5000 configured by the HD 204. The database stored in the storage unit 5000 will be described below.

[0086] [Table 2] The storage unit 5000 has a terminal authentication management DB 5001 having a terminal authentication management table as shown in Table 2. In the terminal authentication management table, each password is associated with each communication ID of all transmission terminals 10 managed by the transmission management device 50. For example, the terminal authentication management table shown in Table 1 indicates that the communication ID of the transmission terminal 10aa is "01aa" and the password is "aaaa."

[0087] [Table 3] The storage unit 5000 also has a terminal management DB 5002 having a terminal management table as shown in Table 3. In the terminal management table, for each communication ID of each transmission terminal 10, the terminal name of each transmission terminal 10 or the information processing terminal 40, the operating status of each transmission terminal 10, and the IP address of the transmission terminal 10 are associated and managed. The terminal name is registered in advance in the transmission management device 50 in association with the communication ID, and the terminal name can be determined when the communication ID is known.

[0088] For example, the terminal management table shown in Table 3(a) indicates that for the transmission terminal 10aa with the communication ID "01aa", the terminal name is "AA terminal, Tokyo office, Japan", the operating status is "online (transmitting)", and the IP address of the transmission terminal 10aa is "1.2.1.3".

[0089] The terminal management table in Table 3(a) shows a state in which the transmission terminal 10aa with the communication ID 01aa, the transmission terminal 10ba with the communication ID 01ba, and the information processing terminal 40bd with the communication ID 04bd are participating in a conference (online (transmitting)). An earlier operating state is online (transmittable).

[0090] Table 3(b) shows the terminal management table when the information processing terminal 40 currently recording a conference is participating in the conference. The operating status of the information processing terminal 40db with the communication ID "04db" is "recording a conference." This operating status of "recording a conference" indicates that the information processing terminal 40bd is recording. The operating status of the information processing terminals 10 and 40 with the communication IDs "01aa" and "01ba" is also "recording a conference." This is because the operating status of the transmitting terminal 10 and the information processing terminal 40 participating in the same conference as the information processing terminal 40 is also changed to "recording a conference" (even if the transmitting terminal 10 and the information processing terminal 40 are not actually recording).

[0091] Here, the communication IDs in the terminal management table are uniquely assigned when the transmission terminals 10 and the information processing terminals 40 are registered in the transmission system 1, and the terminal names are registered or changed by the administrator of the transmission management device 50. Note that the terminal names may be registered or changed in response to a request made from the transmission terminal 10 to the transmission management device 50. That is, the following description will be given assuming that the communication IDs and terminal names are registered in advance.

[0092] Furthermore, the operating status "online (transmitting)" indicates that the transmission terminal 10 is logged in to the transmission system 1 but is not participating in a video conference. "Online (transmitting)" indicates that the transmission terminal 10 is logged in to the transmission system 1 and is participating in a video conference (meaning in a conference). The operating status "offline" indicates that the transmission terminal 10 is not logged in to the transmission system 1. A transmission terminal 10 that is participating in a video conference or in a conference is a transmission terminal 10 whose operating status is "online (transmitting)".

[0093] [Table 4] Furthermore, the storage unit 5000 has constructed a destination list management DB 5003 having a destination list management table as shown in Table 4. In the destination list management table, the communication IDs of all destination terminals registered as candidate destination terminals are associated with the communication ID of the requesting terminal requesting the start of a video conference.

[0094] For example, it is shown that the candidates of destination terminals to which the transmission terminal 10aa with the communication ID "01aa" can request the start of a video conference are the transmission terminals 10ab to 10dc with the communication IDs "01ab" to "01dc" and the information processing terminals 40bd and 40be with the communication IDs "04bd" and "04be". The communication ID of the destination terminal is added when a permission notice (acceptance) is returned from the destination transmission terminal 10 or the information processing terminal 40 in response to a request made from the transmission terminal 10 or the information processing terminal 40 to the transmission management device 50. The communication ID of the destination terminal can also be deleted from the transmission terminal 10 or the information processing terminal 40 to the transmission management device 50.

[0095] [Table 5] The storage unit 5000 also stores a session management DB 5004 configured with a session management table. In this session management table, for each session ID used to execute a session for selecting a relay device 30, the relay device ID of the relay device 30 used to relay image data and audio data, the communication ID of the request source terminal, the communication ID of the destination terminal, and the conference ID are associated and managed. The session ID is an ID for identifying a session, and transmitting terminals 10 participating in the same conference are not necessarily included in the same session. In contrast, the same conference ID is assigned to transmitting terminals 10 participating in the same conference.

[0096] (URI of UI providing device) Furthermore, the storage unit 5000 stores a URI (Uniform Resource Identifier) ​​5005 of the UI providing device 20. The URI 5005 of the UI providing device 20 is set by the transmission management device 50 or an administrator of the UI providing device 20, or the like.

[0097] (Recording management table) The storage unit 5000 stores a recording in progress management table 5006. The recording in progress management table 5006 held by the transmission management device 50 registers all communication IDs of the transmission terminal 10 and the information processing terminal 40 that are recording for each conference.

[0098] (Functional configuration of the transmission management device 50) The transmitting / receiving unit 51 is configured by a CPU 201, a network I / F 2809, etc., and transmits and receives various data and information to and from other transmission terminals 10, relay devices 30, and program providing devices 90 via the transmission network 2.

[0099] The terminal authentication unit 52 is configured by the CPU 101 etc., and searches the terminal authentication management DB 5001 using the communication ID and password included in the login request information received via the transmission / reception unit 51 as search keys. Then, it performs terminal authentication by determining whether the same communication ID and password are managed in the terminal authentication management DB 5001. Note that the authentication method is not limited to this, and a client certificate (an authentication method using a public key and a private key) may also be used.

[0100] The status management unit 53 is configured by the CPU 101 etc., and manages the operating status of the requesting terminal that has made the login request. That is, the operating status and IP address of the requesting terminal are stored and managed in the terminal management DB 5002 in association with the communication ID of the requesting terminal.

[0101] Also, the user turns the state of the power switch 109 of the transmission terminal 10 from ON to OFF. As a result, the state management unit 53 changes the operation state of the transmission terminal 10 in the terminal management DB 5002 from online to offline based on the state information sent from the transmission terminal 10 indicating that the power is to be turned OFF. Also, the state management unit 53 periodically (or irregularly) communicates to monitor the alive status of the transmission terminal 10 and the information processing terminal 40.

[0102] The terminal extraction unit 54 is configured by the CPU 101 etc., and searches the destination list management DB 5003 using the communication ID of the requesting terminal that made the login request as a search key, and extracts the communication IDs of candidate destination terminals for the requesting terminal.

[0103] In addition, the terminal extraction unit 54 searches the destination list management DB 5003 using the communication ID of the requesting terminal that has made the login request as a search key, and also extracts the communication IDs of other requesting terminals that have registered the communication ID of the requesting terminal as a candidate destination terminal.

[0104] The terminal status acquisition unit 55 is composed of the CPU 101 etc., and searches the terminal management DB 5002 using the communication ID of the candidate destination terminal extracted by the terminal extraction unit 54 as a search key, and reads out the operating status for each detected communication ID.

[0105] This allows the terminal status acquisition unit 55 to acquire the operating status of candidate destination terminals for the request source terminal that has made the login request. The terminal status acquisition unit 55 also searches the terminal management DB 5002 using the communication ID extracted by the terminal extraction unit 54 as a search key, and acquires the operating status of the request source terminal that has made the login request.

[0106] When a session is established between the transmitting terminals 10 and 10, between the transmitting terminals 10 and 40, or between the information processing terminals 40 and 40, the session management unit 56 manages the session in a session management table. When another session is established for the transmitting terminal 10 or the information processing terminal 40 with which a session is already established, the session management unit 56 assigns the same conference ID, so that the transmitting terminals 10 and the information processing terminals 40 participating in the same conference are linked by the conference ID.

[0107] The storage / readout processing unit 59 is configured by the CPU 101, the HDD 205, etc., and stores various data in the storage unit 5000 and reads out various data stored in the storage unit 5000.

[0108] <<Functional Configuration of UI Providing Device>> The UI providing device 20 includes a transmission / reception unit 81, a server unit 82, and a storage / readout processing unit 89. Each of these units is a function realized by a CPU 201 that executes a UI providing device program 2000a deployed from the HD 204 onto the RAM 203. Further, the transmission management device 50 includes a storage unit 2000 constituted by the HD 204. Layout information lot is stored in the storage unit 2000.

[0109] (Functional Configuration of UI Providing Device) The transmission / reception unit 81 is constituted by the CPU 201 and a network I / F 2809 or the like, and is configured to transmit and receive various data and information to and from the transmission terminal 10 via the transmission network 2.

[0110] The server unit 82 is constituted by a CPU 101 or the like, acquires a layout information request signal from the transmission terminal 10, and transmits the layout information lot read by the storage / readout processing unit 29 from the storage unit 2000 to the transmission terminal 10 via the transmission / reception unit 81.

[0111] The storage / readout processing unit 89 is constituted by a CPU 101 or the like, and is configured to store various data in the storage unit 2000 or read out various data stored in the storage unit 2000. The storage unit 2000 is constituted by a ROM 202, a RAM 203, an HDD 205, etc., and mainly stores the UI providing device program 2000a and the layout information lot.

[0112] <<Functional Configuration of Information Processing Terminal>> 7 is an example of a functional block diagram of the information processing terminal 40. The functions of the information processing terminal 40 will be described mainly focusing on the differences from the transmitting terminal 10. The functions of the information processing terminal 40 are similar to those of the transmitting terminal 10, but the information processing terminal 40 differs from the transmitting terminal 10 in that it has an image recording unit 41. Each function of the information processing terminal 40 is realized by the CPU 101 that executes the information processing terminal program 1000c expanded from the flash memory 104 onto the RAM 103 shown in FIG. 4. Furthermore, the information processing terminal 40 stores the information processing terminal program 1000c and recorded data 1000d in the storage unit 1000.

[0113] The image recording unit 41 records the image data and audio data displayed on the display 120 as recorded data 1000d. The recorded data is data in which the full screen display screen 501 or the information display screen 601 displayed on the display 120 by the display control unit 17 is recorded. It is possible to save any screen displayed on the display 120, not just the full screen display screen 501 or the information display screen 601. It is also possible to record (record) audio data output from the speaker 115 by the sound output unit 15b. In this embodiment, it is assumed that audio data is also recorded when the image recording unit 41 records. On the other hand, the image recording unit 41 is capable of recording only image data or only audio data.

[0114] In this embodiment, the information processing terminal 40 has a recording function, and the transmission terminal 10 does not have a recording function, but this is merely a distinction for the purpose of explanation, and the transmission terminal 10 may also have a recording function.

[0115] Furthermore, the recorded data 1000d may be stored not in the storage unit 1000 but in a network-attached storage (NAS) or the like on a network.

[0116] <Processing or Operation of the Embodiment> Next, the operation of the transmission system 1 according to this embodiment will be described. FIG. 8 is a sequence diagram showing a preparatory operation until the transmitting terminal 10 is ready to participate in a video conference. Note that FIG. 8 shows an example in which the transmitting terminal 10aa is ready. In the preparatory operation, various information is transmitted and received in the management information session sei. The process in which the information processing terminal 40 participates in a video conference is also the same as that of the transmitting terminal 10.

[0117] First, when the user of the transmission terminal 10aa turns on the power switch 109 of the transmission terminal 10aa, the operation input accepting unit 12 accepts the power-on command and turns on the power (step S21). Then, triggered by the acceptance of the power-on command, the login request unit 13 transmits login request information representing a login request from the transmitting / receiving unit 11 of the transmission terminal 10aa to the transmission management device 50 via the transmission network 2 (step S22).

[0118] The login request information includes a communication ID and a password for identifying the transmission terminal 10aa that has made the login request. The communication ID and the password are read from the storage unit 1000 by the storage / read processing unit 19.

[0119] When the login request information transmitted from the transmission terminal 10aa is received by the transmission management device 50, the transmission management device 50 can grasp the IP address of the transmission terminal 10aa, which is the sender.

[0120] Next, the terminal authentication unit 52 of the transmission management device 50 searches the terminal authentication management DB 5001 in the storage unit 5000 using the communication ID and password included in the login request information received via the transmitting / receiving unit 51 as search keys. Then, the terminal authentication unit 52 performs terminal authentication by determining whether the communication ID and password identical to the search key are managed in the terminal authentication management DB 5001 (step S23).

[0121] Here, there may be a case where the terminal authentication unit 52 determines that the same communication ID and password as the search key are not managed, that is, that the login request is not from a transmission terminal 10 with valid usage authority. In this case, the transmitter / receiver 51 transmits authentication result information indicating the authentication result obtained by the terminal authentication unit 52 to the transmission terminal 10aa that has made the login request via the transmission network 2, and the operation in the preparation stage ends.

[0122] On the other hand, if the terminal authentication unit 52 determines that the same communication ID and password as the search key are managed, that is, if it determines that the login request is from a transmission terminal 10 with valid usage authority, the status management unit 53 associates the communication ID, operating status, and IP address of the transmission terminal 10aa and stores them in the terminal management DB 5002 (step S24).

[0123] As a result, the terminal management table manages the communication ID "01aa," the operating state "online (transmission possible)," and the IP address "1.2.1.3" of the transmission terminal 10aa in association with each other.

[0124] Next, the transmitter / receiver 51 of the transmission management device 50 transmits authentication result information indicating the authentication result obtained by the terminal authentication unit 52 to the transmission terminal 10aa that has requested login via the transmission network 2 (step S25). In addition, the transmission management device 50 transmits the URI 5005 of the UI providing device 20 to the transmission terminal 10 together with or before or after the authentication result information. This allows the transmission terminal 10 to access the UI providing device 20.

[0125] The URI 5005 of this UI providing device 20 indicates the storage location of the layout information lot in the transmission network 2. For example, it has a format such as "IP address (or domain name) + folder name + file name", but it is sufficient if it is accessible to the UI providing device 20.

[0126] Since the URI 5005 is not held by each transmission terminal 10, even if the UI providing device 20 is replaced or the URI is changed, the administrator of the transmission management device 50 only needs to change the URI stored in the transmission management device 50. In other words, since changes are not required for all transmission terminals 10, the management load can be reduced.

[0127] Next, the terminal extraction unit 54 of the transmission management device 50 searches the destination list management DB 5003 using the communication ID “01aa” of the transmission terminal 10aa that has made the login request as a search key, and extracts the communication IDs of candidate destination terminals for the transmission terminal 10aa (step S26).

[0128] In this case, the communication IDs "01ab, 01ac, 01ba, 01bb, 01bc, 04bd, 04be, 01ca, 01cb, 01ca, 01da, 01db, 01dc" are extracted as destination terminals for the communication ID "01aa" of the transmission terminal 10aa.

[0129] Next, the terminal status acquisition unit 55 of the transmission management device 50 searches the terminal management DB 5002 using the communication IDs "01ab, 01ac, 01ba, 01bb, 01bc, 04bd, 04be, 01ca, 01cb, 01ca, 01da, 01db, 01dc" of the destination terminal candidates extracted by the terminal extraction unit 54 as search keys. Then, the terminal status acquisition unit 55 reads out the operation status "offline," "online (transmission possible)," "online (transmitting)," or "recording conference" for each communication ID extracted by the terminal extraction unit 54, thereby acquiring the operation status of each of the transmission terminal 10 and the information processing terminal 40 (step S27).

[0130] Next, the transmitter / receiver 51 of the transmission management device 50 adds the communication IDs "01ab, 01ac, 01ba, 01bb, 01bc, 04bd, 04be, 01ca, 01cb, 01ca, 01da, 01db, 01dc" which are the search keys used in step S27, and the operation states of each transmission terminal 10 and the information processing terminal 40, "offline," "online (transmission possible)," "online (transmitting)," and "recording conference in progress," to the destination state information, and then transmits the destination state information to the transmission terminal 10aa via the transmission network 2 (step S28).

[0131] This allows the transmitting terminal 10aa to grasp the current operating status of the transmitting terminals 10ab, 10ac, 10ba, 10bb, 10bc, 10ca, 10cb, 10ca, 10da, 10db, 10dc, and the information processing terminals 40bd and 40be, which are candidates for the destination terminals, namely "offline," "online (transmission possible)," "online (transmitting)," and "recording conference in progress."

[0132] Furthermore, the terminal extraction unit 54 of the transmission management device 50 searches the destination list management DB 5003 using the communication ID "01aa" of the transmission terminal 10aa that has made the login request as a search key. Then, the terminal extraction unit 54 extracts the communication IDs of other transmission terminals that have registered the communication ID "01aa" of the transmission terminal 10aa as a destination terminal candidate (step S29). According to the destination list management table of Table 4, "01ba", "01ca", and "04bd" are extracted.

[0133] Next, the status management unit 53 of the transmission management device 50 searches the terminal management DB 5002 using the communication ID “01aa” of the transmission terminal 10aa that has made the login request as a search key, and acquires the operating status “online (transmission possible)” of the transmission terminal 10aa that has made the login request (it is online (transmission possible) immediately after login) (step S30).

[0134] Next, the transmitter / receiver 51 of the transmission management device 50 identifies the transmission terminals 10ba and 10ca and the information processing terminal 40db identified by the communication IDs "01ba," "01ca," and "04bd" extracted in step S29, and identifies the transmission terminals 10ba and 40bd whose operating status is "online" in the terminal management DB 5002. This is because destination status information cannot be received unless the terminals are online. Then, the transmitter / receiver 51 transmits destination status information including the communication ID "01aa" and operating status "online (transmission possible)" of the transmission terminal 10aa acquired in step S30 (steps S31-1 and S31-2).

[0135] The destination list display unit 16 of the transmission terminal 10aa uses the destination status information received in step S28 to display a destination list screen on the display 120aa (S32). The destination list screen is shown in FIG.

[0136] When transmitting destination status information to the transmission terminal 10ba and the information processing terminal 40bd, the transmission / reception unit 51 refers to the IP addresses of the transmission terminals managed in the terminal management table based on the respective communication IDs "01ba" and "04bd".

[0137] This allows the communication ID "01aa" and operating status "online (transmission possible)" of the transmission terminal 10aa that has made the login request to be communicated to the transmission terminal 10db and the information processing terminal 40bd, which can transmit to the transmission terminal 10aa that has made the login request.

[0138] Furthermore, when step S28 of the preparation stage operation is completed, the display control unit 17 of the transmitting terminal 10aa highlights the terminals participating in the video conference on the display 120 from the destination list screen displayed on the display 120. That is, by highlighting the information of the transmitting terminal 10ba and the information processing terminal 40bd whose operating status is "online (transmitting)" or "online (transmittable)" on the display 120, the user can recognize the transmitting terminals 10 participating in the video conference or the transmitting terminals 10 that can participate.

[0139] 9 is a sequence diagram showing a conference participation operation until the transmission terminal 10 enters a participation state for participating in a video conference. In FIG. 9, an example is shown in which the transmission terminal 10aa holds a video conference with the information processing terminal 40bd. In the conference participation operation, various information is transmitted and received through the management information session sei.

[0140] First, when the user presses the operation button 108 to select, for example, the information processing terminal 40bd to select participation in a video conference, the operation input accepting unit 12 accepts a request to participate in the video conference with the information processing terminal 40bd (step S41). Note that, when the transmission terminal 10aa is the information processing terminal 40, the request to participate in the video conference is accepted by pressing a hard key or a soft key displayed on the touch panel of the information processing terminal 40 instead of the operation button 108.

[0141] In response to this request, the transmitter / receiver unit 11 of the transmission terminal 10aa transmits participation request information to the transmission management device 50, which includes the communication ID "01aa" of the transmission terminal 10aa and the communication ID "04bd" of the information processing terminal 40bd, and indicates that the transmission terminal 10aa will participate in the video conference (step S42).

[0142] As a result, the transmitting / receiving unit 51 of the transmission management device 50 can receive the participation request information and confirm the IP address "1.2.1.3" of the transmission terminal 10aa that is the sender.

[0143] Next, the status management unit 53 of the transmission management device 50 uses the communication ID "01aa" of the transmission terminal 10aa included in the participation request information as a search key. Then, the status management unit 53 searches the terminal management DB 5002 and sets the operation status of the transmission terminal 10aa that has requested participation to "online (transmitting)" (step S43). As a result, the operation status in the terminal management table is updated as shown in Table 3(a).

[0144] Next, the transmitter / receiver 51 of the transmission management device 50 transmits start request information including the communication ID “01aa” of the transmission terminal 10aa to the information processing terminal 40bd via the transmission network 2 (step S44). This allows the information processing terminal 40bd to know which transmission terminal 10 has requested to participate in the video conference.

[0145] Next, the information processing terminal 40bd transmits reception completion information indicating that reception of the start request information has been completed to the transmission management device 50 from the transmitting / receiving unit 11 via the transmission network 2 (step S45). The session management unit 56 assigns (assigns) a unique session ID to the session between the transmission terminal 10aa, which is the request source, and the information processing terminal 40bd, which is the destination terminal. In addition, the session management unit 56 assigns a unique conference ID and registers the session ID, relay device ID, request source communication ID, destination communication ID, and conference ID in the session management table.

[0146] Next, the transmitter / receiver 51 of the transmission management device 50 transmits relay start request information representing a request to start relaying to the relay device 30 via the transmission network 2 (step S46). The relay start request information includes the IP addresses “1.2.1.3” and “1.3.1.3” of the transmission terminal 10aa and the information processing terminal 40bd, the session ID, and the conference ID.

[0147] As a result, the relay device 30 establishes a session for transmitting three types of image data (low resolution, medium resolution, and high resolution) and audio data between the transmitting terminal 10aa and the information processing terminal 40bd (step S47). In this way, the transmitting terminal 10aa can participate in the video conference with the information processing terminal 40bd.

[0148] 9, when a start request is sent to the transmission management device 50 for the transmission terminal 10 or the information processing terminal 40 communicating in the same session, the session management unit 56 of the transmission management device 50 assigns the conference ID of the transmission terminal 10 and the information processing terminal 40 that are starting the session. Therefore, even if the session ID changes, the same conference ID is assigned to the transmission terminal 10 or the information processing terminal 40 participating in the same conference. The relay device 30 transfers image data and audio data to the transmission terminal 10 or the information processing terminal 40 associated with the same conference ID.

[0149] Meanwhile, the transmitter / receiver 51 of the transmission management device 50 transmits terminal status change information (operation status change information) including the communication ID "01aa" and the operation status "online (transmitting)" of the transmission terminal 10aa to each transmission terminal 10 registered in the terminal management DB 5002 (step S48). As a result, each transmission terminal that receives the terminal status change information can store and display the operation status of the transmission terminal 10aa with the communication ID "01aa" as necessary.

[0150] By establishing a session, the transmitting terminal 10aa can display image data and output audio data transmitted from the information processing terminal 40bd. The information processing terminal 40bd can display image data and output audio data transmitted from the transmitting terminal 10aa. Even if there are three or more transmitting terminals 10 participating in the video conference, the image data and audio data are transmitted in the same manner as when there are two transmitting terminals. Alternatively, the image data and audio data may be transmitted using a communication protocol that does not involve the relay device 30 when there are two transmitting terminals, but does involve the relay device 30 when there are three or more transmitting terminals.

[0151] <Recording operations> The operation when the information processing terminal 40 performs recording will be described with reference to Fig. 10. Fig. 10 is an example of a sequence diagram showing the operation when the information processing terminal 40bd starts and ends recording.

[0152] S1: A user (conference participant) of the information processing terminal 40bd during a video conference performs an operation to record image data. The operation input accepting unit 12 of the information processing terminal 40bd accepts this operation. First, the transmitting / receiving unit 11 of the information processing terminal 40bd transmits a recording start notification to the transmission management device 50 together with the communication ID (04bd) of the information processing terminal 40bd.

[0153] S2: The transmitter / receiver 51 of the transmission management device 50 receives the recording start notification. As a result, the status management unit 53 sets the operating status associated with the communication ID (04bd) of the information processing terminal 40bd in the terminal management table to “recording conference in progress.”

[0154] Furthermore, the state management unit 53 sets the operating states of all the transmitting terminals 10 and the information processing terminals 40 that are holding a video conference with the information processing terminal 40bd to "recording conference in progress." This is because there is a risk that the participants and materials of all the transmitting terminals 10 and the information processing terminals 40 participating in the same conference may be recorded, and the participants of the transmitting terminals 10 and the information processing terminals 40 who will soon participate in the video conference between the transmitting terminal 10aa and the information processing terminal 40bd are to be notified that a destination terminal is participating in the conference in which image data and audio data are being recorded.

[0155] The transmitting terminal 10 and the information processing terminal 40 that are holding a video conference with the information processing terminal 40bd are registered in the session management table. The terminal extraction unit 54 extracts the conference ID associated with the communication ID (04bd) of the information processing terminal 40bd in the session management table, and further acquires all communication IDs associated with this conference ID. In the session management table of Table 5, the transmitting terminals 10aa and 10ba with the communication ID 01aa are participating in the same conference as the information processing terminal 40bd. Therefore, the status management unit 53 sets the operating status of the information processing terminal 40bd and the transmitting terminals 10aa and 10ba to "recording conference in progress."

[0156] The state management unit 53 registers the communication ID (04bd) of the information processing terminal 40bd in the recording management table 5006. The communication IDs of the transmission terminals 10aa and 10ba do not need to be registered (because they are not recording).

[0157] S3: The transmitter / receiver 51 of the transmission management device 50 transmits a recording start notification to the transmitting terminal 10 and the information processing terminal 40 that are holding a video conference with the information processing terminal 40bd. The method of determining the transmitting terminal 10 and the information processing terminal 40 that are holding a video conference with the information processing terminal 40bd is the same as in step S2. The information processing terminal 40bd and the transmitting terminals 10aa and 10ba are identified, but since the transmitting terminal 10ba is omitted in FIG. 10, the transmitter / receiver 51 of the transmission management device 50 transmits a recording start notification to the transmitting terminal 10aa.

[0158] S4: The transmitter / receiver 11 of the transmission terminal 10aa receives the recording start notification, and the display control unit 17 displays the recording notification. Details will be described with reference to FIG.

[0159] S5: When the operation input receiving unit 12 receives an operation to start recording, the image recording unit 41 of the information processing terminal 40bd starts recording image data and audio data.

[0160] S6: Next, the user (conference participant) of the information processing terminal 40bd performs an operation to end recording of the image data. The operation input accepting unit 12 of the information processing terminal 40bd accepts this operation. Note that the image recording unit 41 of the information processing terminal 40bd stops recording of the image data and audio data when the operation input accepting unit 12 accepts the operation to end recording.

[0161] S7: The transmitting / receiving unit 11 of the information processing terminal 40bd transmits a recording end notification (an example of a recording end notification) to the transmission management device 50 together with the communication ID (04bd) of the information processing terminal 40bd.

[0162] S8: The transmitter / receiver 51 of the transmission management device 50 receives the recording end notification. The status management unit 53 deletes the communication ID (04bd) of the information processing terminal 40bd from the recording in progress management table 5006. Then, if there is no communication ID in the recording in progress management table 5006, the status "recording conference in progress" in the terminal management table is set to "online (transmitting)." By doing so, if even one transmitting terminal 10 or information processing terminal 40 remains in the recording progress state, the operating status of all transmitting terminals 10 or information processing terminals 40 participating in the conference can remain "recording conference in progress."

[0163] S9: The transmitter / receiver 51 of the transmission management device 50 transmits a recording end notification to the transmitting terminal 10 and the information processing terminal 40 that are holding a video conference with the information processing terminal 40bd. Therefore, the recording end notification is transmitted to the transmitting terminal 10aa (the transmitting terminal 10ba is omitted in FIG. 10).

[0164] S10: The transmitter / receiver 11 of the transmission terminal 10aa receives the recording end notification, and the display controller 17 hides the recording notification. Details will be described with reference to FIG.

[0165] The recording end notification is sent when the user performs an operation to end recording, but the recording end notification may also be sent when an indirect operation to end recording (for example, when the power of the information processing terminal 40bd is turned off) is performed.

[0166] Furthermore, the information processing terminal 40bd may transmit a recording start notification and a recording end notification directly to the transmission terminal 10aa without going through the transmission management device 50. Furthermore, a recording start notification is sent at the start of recording, but after recording has started, a notification such as a recording in progress notification may be sent periodically or irregularly during the start of recording.

[0167] <Display example> The screen displayed on the display 120 by the transmission terminal 10 will be described.

[0168] <<Destination list screen>> FIG. 11 shows an example of a destination list screen displayed in step S32 of FIG. 8. The destination list screen displays the operating status and terminal name based on the destination status information transmitted in step S28. For convenience of explanation, the destination list screen displayed by the transmission terminal 10aa based on the terminal management table of Table 3(b) will be described here. Note that in step S28 of FIG. 8, the transmission terminal 10aa is about to participate in a video conference, so the terminal management table of Table 3(b) does not match the processing content. FIG. 11 is a diagram for explaining that the destination list screen is generated based on the terminal management table.

[0169] In the destination list management table, the communication IDs of the destination terminals of the transmitting terminal 10aa are "01ab, 01ac, 01ba, 01bb, 01bc, 04bd, 04be, 01ca, 01cb, 01ca, 01da, 01db, 01dc." Of these, the information processing terminal 40bd with the communication ID 04bd and the transmitting terminal 10ba with the communication ID 01ba are currently recording the conference, so the destination list display unit 16 displays the recording location mark 71 in the operating status of the communication IDs 04bd and 01ba. The other transmitting terminals 10 and the information processing terminal 40 are offline, so the destination list display unit 16 displays the offline mark in the operating status.

[0170] In this way, each of the destination candidates displays its operating status (recording location mark 71), so users who are about to join the video conference know that they and their materials will be recorded if they join, and can therefore decide not to participate.

[0171] 12 is an example of a diagram illustrating a confirmation dialog screen 510 that is displayed when the user selects a destination with a recording location mark 71. When the user selects a destination with a recording location mark 71 displayed, the operation input accepting unit 12 accepts the selection. When a destination with a recording location mark 71 displayed is selected, the destination list display unit 16 displays the confirmation dialog screen 510.

[0172] A confirmation dialog screen 510 displays a message 511 saying "You will join a video conference currently being recorded. Is this OK?", an OK button 512, and a Cancel button 513. When the user presses the OK button 512, the start request information of step S41 in Fig. 9 is sent, and when the user presses the Cancel button 513, the start request information is not sent. Therefore, the user can confirm in advance before joining a video conference currently being recorded.

[0173] <<Screen during the meeting>> In this embodiment, the display control unit 17 has two display modes: a full screen display mode and an information display mode.

[0174] 13 is a diagram showing an example of a screen displayed on the display 120 in full-screen display mode. This screen is called full-screen display screen 501. The full-screen display mode is displayed when a session is established in step S47 of FIG. 9 and transmission or reception of image data and audio data begins. In other words, of the two display modes, this is the display mode that is primarily used during a video conference.

[0175] In the full-screen display mode, the display control unit 17 of the transmitting terminal 10 displays all images transmitted from each transmitting terminal 10 participating in the video conference on the display 120. If displaying all images results in a small image area, a predetermined upper limit (maximum value) of images is displayed.

[0176] 13 is assigned one relatively large first image area 502 and multiple relatively small second image areas 503. For example, an image represented by image data transmitted from each transmitting terminal 10 participating in the video conference is displayed in each second image area 503, and an image represented by image data of a main transmitting terminal 10 or information processing terminal 40 among the transmitting terminals 10 and information processing terminals 40 participating in the video conference is displayed in the first image area 502.

[0177] For example, the display control unit 17 identifies, among the transmitting terminals 10 participating in the video conference, the transmitting terminals that have spoken in the received audio data as the main terminals. The transmitting terminals 10 that have spoken are sorted in descending order of volume. The display control unit 17 allocates and displays images in the first image area 502 in this order.

[0178] Note that the full-screen display screen 501 shown in FIG. 13 is just one example, and in full-screen display mode, it is possible to display each image with all multiple image areas of the same size, or to display only the image of the main terminal mentioned above across the entire display 120.

[0179] Full-screen display screen 501 has auxiliary area 173. In full-screen display mode, auxiliary area display unit 18a can display auxiliary area 173. In Fig. 13, auxiliary area 173 is a strip-shaped blank space at the top of full-screen display screen 501, but it may also be a strip-shaped blank space at the right edge, left edge, or bottom, or it may be displayed so as to overlap with the image.

[0180] The auxiliary area 173 is divided into a first area ar1 to a sixth area ar6. The first area ar1 displays the communication ID of the own terminal. A user of the transmitting terminal 10 who wishes to participate in the same video conference as this transmitting terminal 10 can participate in the video conference by inputting or selecting this communication ID into the transmitting terminal 10 that is currently inquiring. Second area ar2...The number of locations is displayed. Third area ar3...displays the conference time since the session was established. Fourth area ar4...Optional information is displayed. Fifth area ar5...Network bandwidth information (upstream and downstream communication speeds) is displayed. The sixth area ar6... displays the cursor button. The cursor button 1736 has an upper triangle button 1736u and a lower triangle button 1736d, with the upper triangle button 1736u being black and the lower triangle button 1736d being white. The upper triangle button 1736u indicates that it can be operated, while the lower triangle button 1736d indicates that it cannot be operated. When the upper triangle button 1736u is operated, the simplified display auxiliary area 173, which will be described later, is displayed.

[0181] It is preferable that the auxiliary area 173 is displayed semi-transparently, which makes it easier to maintain visibility even if the auxiliary area 173 overlaps with image data transmitted between locations.

[0182] Here, the cursor button 1736 in the sixth area ar6 is an icon that notifies the user that it is possible to switch to the simplified display auxiliary area 173. Hereinafter, the auxiliary area in Fig. 13 will be referred to as the "standard auxiliary area" to distinguish it from the "simplified display auxiliary area" described later.

[0183] The cursor button 1736 corresponds to two of the several operation buttons 108. It is preferable that the corresponding operation button 108 can be swung up and down, and the upward operation of the operation button 108 corresponds to the upper triangle button 1736u, and the downward operation of the operation button 108 corresponds to the lower triangle button 1736d.

[0184] Furthermore, upon receiving a recording start notification, the auxiliary area display unit 18a expands the auxiliary area 173 to display the recording notification information 80. It is preferable that the recording notification information 80 is also displayed semi-transparently. In FIG. 13, the lower right portion of the auxiliary area 173 is expanded, and the other location recording mark 72 is displayed in the recording notification information 80. In addition, a label 73 with the words "Recording at other location" is displayed to the right of the other location recording mark 72. It is sufficient that at least one of the other location recording mark 72 and the label 73 is displayed. Participants can understand from the other location recording mark 72 or the label 73 that image data is being recorded at another location. As is clear from the processing in FIG. 10, if one or more of the participating locations are recording, the auxiliary area display unit 18a displays the recording notification information 80.

[0185] Although the illustrated remote location recording mark 72 is a red or orange circle, the appearance (color, shape, size, etc.) of the remote location recording mark 72 may be, for example, a polygon such as a triangle or a square, and the color may be blue or green. The size may also be smaller or larger than shown. The location of the recording notification information 80 may be the lower right portion, the lower left portion, the upper right portion, the upper left portion, or above or below the center of the auxiliary area 173, in addition to the lower right portion. If there is space in the auxiliary area 173, at least one of the remote location recording mark 72 and the label 73 may be placed in that space.

[0186] Fig. 14 is a diagram showing an example of the simplified display auxiliary area 173. In Fig. 14, the number of locations is the same as in Fig. 13, but information on the transmission status in the first area ar1 to the sixth area ar6 is displayed in a simplified manner. In the simplified display auxiliary area 173, information on what is displayed in each area is not displayed, and mainly, only the communication ID value is displayed in the first area ar1, only the number of locations in the second area ar2, only the time in the third area ar3, and only the bandwidth in the fifth area ar5.

[0187] The display form of the cursor button 1736 in the sixth area ar6 is different from that in the standard auxiliary area 173. In the simplified display auxiliary area 173, both the upper triangle button 1736u and the lower triangle button 1736d are black. The user can operate both the operation button 108 corresponding to the upper triangle button 1736u and the operation button 108 corresponding to the lower triangle button 1736d.

[0188] By operating the cursor button 1736, the display of the auxiliary area 173 is controlled as follows. Bottom triangle button: Switch to standard auxiliary area 173. Upper triangle button: Do not display the simplified display auxiliary area 173 (do not display the standard auxiliary area 173 either).

[0189] The simplified display auxiliary area 173 occupies less area of ​​the display 120 than the standard auxiliary area 173, so the visibility of the image data can be improved while still displaying information about the transmission status in the auxiliary area 173.

[0190] It is preferable that the simplified display auxiliary area 173 is displayed semi-transparently, similar to the standard auxiliary area 173.

[0191] The display mode of the other-location recording mark 72 or label 73 in the simplified display auxiliary area 173 is the same as in the standard display auxiliary area 173. However, the auxiliary area display unit 18a may change the display mode, such as by hiding either the other-location recording mark 72 or the label 73. In this way, the recording notification information 80 is always displayed in both the standard display auxiliary area 173 and the simplified display auxiliary area 173, allowing participants to grasp the recording status in real time.

[0192] 15 is a diagram showing an example of how the other-location recording mark 72 and label 73 are displayed when the auxiliary area 173 is not displayed. Even when the auxiliary area 173 is not displayed, the auxiliary area display unit 18a displays the recording notification information 80. That is, whether or not another location is recording is important information, so the auxiliary area display unit 18a always displays the recording notification information 80 even when the auxiliary area 173 is not displayed. Therefore, participants can grasp the recording status in real time.

[0193] FIG. 16 shows an example of the full-screen display screen 501 when none of the participating locations are recording. Note that FIG. 16 describes the case where the standard auxiliary area 173 is displayed. As shown in FIG. 16, when there are no more locations recording, the auxiliary area display unit 18a hides (deletes) the recording notification information 80. Therefore, participants can know in real time that there are no more locations recording. Note that the same applies to the simplified display auxiliary area 173, where the recording notification information 80 is not displayed. If the auxiliary area 173 is not displayed, the recording notification information 80 is not displayed either.

[0194] <<Display control of recording notification information during a meeting>> 17 is an example of a flowchart illustrating a procedure in which the transmitting terminal 10aa during a conference controls the display of the recording notification information 80. Since the recording notification information 80 is always displayed on the full-screen display screen 501, the procedure in FIG. 17 does not require the auxiliary area to be displayed.

[0195] First, the auxiliary area display unit 18a determines whether or not a recording start notification or a recording end notification has been received (S10).

[0196] When the recording start notification is received, the auxiliary area display unit 18a adds the communication ID transmitted together with the recording start notification to the recording in progress management table 1000b (S20). Therefore, the recording in progress management table 1000b records all the communication IDs of the transmitting terminal 10 and the information processing terminal 40 that are currently recording.

[0197] The auxiliary area display unit 18a determines whether or not the recording notification information 80 is already displayed (S30). The case where the recording notification information 80 is already displayed means that after the transmitting terminal 10 or the information processing terminal 40 participating in the video conference has transmitted a recording start notification, another transmitting terminal 10 or information processing terminal 40 has transmitted a recording start notification.

[0198] If the determination in step S30 is Yes, the recording notification information 80 is already being displayed, so the process in FIG. 17 ends.

[0199] If the determination in step S30 is No, the auxiliary area display unit 18a displays the recording notification information 80 (S40).

[0200] When the recording end notification is received in step S10, the auxiliary area display unit 18a deletes the communication IDs transmitted together with the recording start notification from the recording in progress management table 1000b (S50). Therefore, only the communication IDs of the transmitting terminal 10 and the information processing terminal 40 that are continuing recording are recorded in the recording in progress management table 1000b.

[0201] The auxiliary area display unit 18a determines whether or not the recording in progress management table 1000b has no communication ID registered therein (S60).

[0202] If the determination in step S60 is No, there is no need to hide the recording notification information 80 because there are the transmitting terminal 10 and the information processing terminal 40 that are continuing to record, and the processing in FIG. 17 ends.

[0203] If the determination in step S60 is Yes, there is no transmitting terminal 10 or information processing terminal 40 continuing recording, and therefore the auxiliary area display unit 18a does not display the recording notification information 80 (S70).

[0204] By this procedure, if the transmitting terminal 10 and the information processing terminal 40 that have started recording are present, the recording notification information 80 can be displayed in real time, and if the transmitting terminal 10 and the information processing terminal 40 that are continuing recording are no longer present, the recording notification information 80 can be hidden. Also, if at least one communication ID is stored in the recording in progress management table 1000b, the recording notification information 80 can always be displayed.

[0205] <<If a network disconnection occurs>> During a conference, a session may be interrupted due to a decrease in network bandwidth, etc. Or, a participant may accidentally unplug a cable required for communication. If the transmission terminal 10 or information processing terminal 40 whose session has been interrupted is recording, the transmission terminals 10 or information processing terminals 40 at other locations will continue to display the recording notification information 80. For this reason, it is preferable that the transmission management device 50 detects the transmission terminal 10 or information processing terminal 40 whose session has been interrupted and notifies the transmission terminals 10 or information processing terminals 40 at other locations of the interruption, even if no recording end information is transmitted.

[0206] 18 is an example of a sequence diagram showing the operation procedure of the transmission system 1 when the transmission management device 50 detects that a session has been disconnected. Note that the processing from steps S1 to S5 in FIG. 18 is the same as that in FIG.

[0207] S6: The session management unit 56 of the transmission management device 50 periodically monitors the alive status of the transmission terminal 10 and the information processing terminal 40 with which the session has been established. That is, it requests a Ping response or the like from the transmission terminal 10 and the information processing terminal 40 with the communication IDs registered in the session management table.

[0208] S7: If the communication between the information processing terminal 40 and the transmission management device 50 is cut off for some reason, the session management unit 56 can detect that the session has been cut off.

[0209] S8: When the session management unit 56 detects that the session has been interrupted, the status management unit 53 changes the operation status of the communication ID of the information processing terminal 40bd whose session has been interrupted to offline. The status management unit 53 also deletes the communication ID of the information processing terminal 40bd from the recording in progress management table 5006 and updates the operation status in the terminal management table in the same way as when a recording end notification is received.

[0210] S9: The transmitter / receiver 51 of the transmission management device 50 also transmits operating status change information of the participating locations to the transmitting terminal 10 and the information processing terminal 40 participating in the conference. The terminal status change information includes the communication ID and operating status of the information processing terminal 40bd.

[0211] S10: The transmitter / receiver 11 of the transmitting terminal 10aa receives the terminal status change information and controls the display of the recording notification information 80. That is, if the communication IDs of the transmitting terminal 10 and the information processing terminal 40 other than the information processing terminal 40bd whose session has been interrupted are not registered in the recording in progress management table 1000b, the recording notification information 80 is hidden.

[0212] The process of step S10 in Fig. 18 will be described with reference to Fig. 19. Fig. 19 is an example of a flowchart illustrating the process when the transmission terminal 10aa receives terminal status change information. The process in Fig. 19 is repeatedly executed during the video conference.

[0213] The transmitting / receiving unit 11 of the transmission terminal 10aa determines whether or not the terminal state change information has been received from the transmission management device 50 (S10).

[0214] If the determination in step S10 is Yes, the auxiliary area display unit 18a of the transmitting terminal 10aa checks the transmitting terminals 10 and information processing terminals 40 that have left the room (become offline) (S20). That is, since the transmitting terminal 10aa has received the destination status information and the terminal status change information, the transmitting terminal 10aa refers to the operation status of each transmitting terminal 10 or information processing terminal 40 in the destination status information and terminal status change information that it received most recently, and compares this with the terminal status change information received in step S10. Through this comparison, the communication ID of each transmitting terminal 10 or information processing terminal 40 whose operation status has changed from "recording conference" to "offline" is identified.

[0215] Then, the auxiliary area display unit 18a determines whether the identified communication ID exists in the recording location ID table (S30).

[0216] If the communication ID does not exist in the recording location ID table (No in S30), the information processing terminal 40bd whose session has been interrupted has not been recording, and the processing in FIG. 19 ends.

[0217] If the communication ID exists in the recording location ID table (Yes in S30), the information processing terminal 40bd whose session was interrupted was recording, so the auxiliary area display unit 18a deletes the communication ID included in the terminal status change information from the recording management table 1000b (S40).

[0218] The auxiliary area display unit 18a determines whether or not the recording in progress management table 1000b has no communication ID registered therein (S50).

[0219] If the determination in step S50 is No, there is no need to hide the recording notification information 80 because there are the transmitting terminal 10 and the information processing terminal 40 that are continuing to record, and the processing in FIG. 19 ends.

[0220] If the determination in step S50 is Yes, there is no transmitting terminal 10 or information processing terminal 40 continuing recording, and therefore the auxiliary area display unit 18a does not display the recording notification information 80 (S70).

[0221] By this procedure, even if there is a change in the participating locations, the transmission terminal 10aa can update the recording notification information 80 in real time.

[0222] <Recording notification information on the information display screen> The recording location mark 71 displayed on the information display screen 601 will be described. The user can alternately switch between two display modes by, for example, operating the operation button 108. Note that when the transmission terminal 10 is the information processing terminal 40, the two display modes can be alternately switched by pressing a hard key or a soft key displayed on a touch panel of the information processing terminal 40 instead of the operation button 108.

[0223] Fig. 20 is a diagram showing an example of an information display screen 601 displayed on the display 120 in the information display mode. Fig. 20 shows the information display screen 601 displayed by the transmitting terminal 10aa. As shown in Fig. 20, in the information display mode, the location list display unit 18b of the transmitting terminal 10 displays terminal information including a serial number 614, a terminal name 615, and a communication ID 616 of each transmitting terminal 10 participating in the video conference on the display 120. The serial number 614 is a number assigned to each terminal based on the order of participation, and the largest number in the serial number 614 indicates the number of locations (terminals) participating.

[0224] Also, the information processing terminal 40bd displays a recording location mark 71. As will be described later, the location list display unit 18b displays the recording location mark 71 in front of the serial numbers of the transmitting terminals 10 and information processing terminals 40 whose operating status is recording a conference, respectively. The display manner of the recording location mark 71 is not limited to that shown in the figure, similar to the other location recording mark 72.

[0225] 20 has an image display area 520 for displaying an image and a terminal information display area 530 for displaying terminal information. For example, the terminal information display area 530 displays a list of terminal information of the transmitting terminals 10 participating in the video conference.

[0226] Among the terminal information displayed in the list, terminals for which the terminal name is not displayed represent transmission terminals 10 that are not registered as destination terminal candidates for the own terminal in the destination list management DB 5003. For example, the transmission terminal 10db with the communication ID 01db is registered in the destination list management DB 5003 as a destination terminal candidate for the transmission terminal 10aa, as shown in Table 3, and therefore the terminal name is displayed. The transmission terminal 10cd with the communication ID 01cd is not registered in the destination list management DB 5003 as a destination terminal candidate for the transmission terminal 10aa, and therefore the terminal name is not displayed.

[0227] An image represented by image data of a major transmitting terminal among the transmitting terminals 10 participating in the video conference is displayed in the image display area 520. For example, image data of the transmitting terminal 10 or the information processing terminal 40 having the loudest audio data volume is displayed.

[0228] 20, if the locations participating in the conference cannot be displayed on one screen, a scroll bar 613 is displayed. The user can display information about a location hidden outside the screen of the information display screen 601 by operating the operation button 108. As a result, if a location hidden outside the screen is recording, a recording location mark 71 is displayed, and the participant can identify the transmission terminal 10 or the information processing terminal 40 that is recording.

[0229] (Display procedure for information display screen 601) 21 is a sequence diagram showing a terminal information display operation for displaying terminal information in the terminal information display area 530 when the transmission terminal 10aa is switched to the information display mode. In the terminal information display operation shown in FIG. 21, various types of information are transmitted and received through the management information session sei.

[0230] First, the transmission terminal 10aa transmits a terminal information request signal, which indicates a request for terminal information of the transmission terminals and the information processing terminal 40 participating in the video conference, to the transmission management device 50 (step S70). That is, when the user operates the operation button 108 to switch from the full-screen display mode to the information display mode, the operation input receiving unit 12 receives this operation and transmits the terminal information request signal to the transmission management device 50.

[0231] The terminal status acquisition unit 55 of the transmission management device 50 that has received the terminal information request signal searches the terminal management DB 5002 in the storage unit 5000 using the IP address of the sender of the terminal information request signal as a search key, and acquires the communication ID of the transmission terminal 10aa that sent the terminal information request signal (step S71). Note that the transmission terminal 10aa may send the communication ID.

[0232] Next, the terminal status acquisition unit 55 refers to the session management table to identify the communication IDs of the transmitting terminal 10 and the information processing terminal 40 participating in the same conference as the transmitting terminal 10aa. Next, the terminal status acquisition unit 55 searches the terminal management DB 5002 to search for communication IDs and terminal names whose operating status is "online (transmitting)" or "recording conference" (step S72).

[0233] Next, the terminal status acquisition unit 55 searches the destination list management DB 5003 in the memory unit 5000 using the communication ID detected in step S71 (in this embodiment, "01aa") as a search key, and acquires the communication IDs of the transmission terminal 10 and the information processing terminal 40 that have the sender of the terminal information request signal as the destination terminal (step S73).

[0234] Next, the terminal status acquisition unit 55 determines whether each communication ID acquired in step S72 is included in the communication ID (01aa) of the sender of the terminal information request signal or the communication ID acquired in step S73. If it is determined that the communication ID and the terminal name are included, the transmission / reception unit 51 responds with the communication ID and the terminal name to the sender of the terminal information request signal, and if it is determined that the communication ID and the terminal name are not included, the transmission / reception unit 51 responds with only the communication ID to the sender of the terminal information request signal (step S74).

[0235] The location list display unit 18b of the display control unit 17 of the transmitting terminal 10 that has received this response displays the response content in the terminal information display area 530, as in the information display screen 601 shown in FIG. 20 (step S75). First, a consecutive number is assigned to each pair of the terminal name (there may be no terminal name) and the communication ID. Next, all communication IDs in the recording in progress management table 1000b are acquired. Then, the communication ID in the recording in progress management table 1000b is read, and a recording location mark 71 is placed in association with the transmitting terminal 10 or the information processing terminal 40 of this communication ID.

[0236] By such processing, the recording location mark 71 can be displayed on the information display screen 601. In addition to displaying the recording location mark 71, or without displaying the recording location mark 71, the transmitting terminal 10aa may read out the name of the terminal currently recording by voice. In this case, the transmitting terminal 10aa reads out the name of the terminal currently recording by voice at the timing when a recording start notification is transmitted. Alternatively, the transmitting terminal 10aa reads out the name of the terminal currently recording by voice at any timing when the user operates the transmitting terminal 10aa.

[0237] In addition, if the transmitting terminal 10aa receives a recording start notification, a recording end notification, or terminal status change information while the information display screen 601 is being displayed, the location list display unit 18b of the display control unit 17 displays or hides the recording location mark 71 in real time.

[0238] Furthermore, the information display screen 601 may display not only the recording location mark 71 but also the label 73. Furthermore, the information display screen 601 may display only the label 73.

[0239] Therefore, while the transmitting terminal 10aa is holding a video conference with multiple locations, the user of the transmitting terminal 10aa can determine whether recording is in progress for each location (the transmitting terminal 10 and the information processing terminal 40). The recording location mark 71 is updated in real time, so the user can determine the location where recording has started soon after recording begins. The user can know that there is a location where recording is in progress on both the full-screen display screen 501 and the information display screen 601. [Example]

[0240] In this embodiment, an electronic whiteboard 8 equipped with the functions of a transmission terminal 10 will be described. Electronic whiteboards 8 capable of drawing stroke images such as letters, numbers, and figures on a background image displayed on a large display have been known. Electronic whiteboards 8 are widely used in companies, educational institutions, government agencies, and other organizations.

[0241] The electronic whiteboard 8 may have a function to superimpose an externally input image on the display. Also, it is being considered to equip the electronic whiteboard 8 with the functions of the transmission terminal 10. When the electronic whiteboard 8 is equipped with the functions of the transmission terminal 10, it becomes possible to transmit handwritten stroke images on the electronic whiteboard 8 to the transmission terminal 10 at another location.

[0242] 22 is an example of a diagram illustrating an electronic whiteboard 8 having the functions of the transmission terminal 10. The electronic whiteboard 8 has the functions of a general electronic whiteboard and the functions of the transmission terminal 10, and can hold a video conference with the transmission terminal 10dd. The electronic whiteboard 8 switches between displaying a stroke image of an electronic whiteboard application and an image of a video conference application.

[0243] However, in the case of the electronic whiteboard 8, most of the display 3 is occupied by stroke images, making it difficult to see changes in the status of a running but not displayed application, such as a video conferencing application. This poses a problem in that it is difficult for user U2 to notice changes in the status of the video conferencing application.

[0244] For example, a videoconferencing application transmits a stroke image generated by an electronic whiteboard application of the electronic whiteboard 8 to the transmission terminal 10dd. A user U1 of the transmission terminal 10dd can start recording as in the first embodiment. However, even if the videoconferencing application of the electronic whiteboard 8 receives a recording start notification from another location, a user U2 of the electronic whiteboard 8 may not notice this. Therefore, there is a risk that private information may be recorded without the user realizing it.

[0245] 22, "Confidential Information" handwritten on the electronic whiteboard 8 is shared with the transmission terminal 10dd. Because user U1 of the transmission terminal 10dd is recording image data, the video conferencing app on the electronic whiteboard 8 is displaying a recording location mark 71 or a recording location mark 72, but user U2 cannot see it. Recording such confidential information may pose, for example, the risk of insider activity or incidents, or may cause some inconvenience in communication.

[0246] Therefore, in this embodiment, a transmission system 1 is described in which, when a recording start notification is sent to an electronic whiteboard 8 having the functions of a transmission terminal 10, the electronic whiteboard application displays information from a video conferencing application 710 on a display 3, thereby making the user U2 aware of the notification.

[0247] <Outline of operation of the electronic whiteboard 8> Fig. 23 is an example of a diagram for explaining an outline of the operation of the electronic whiteboard 8 of this embodiment. As shown in Fig. 23(a), the electronic whiteboard 8 has a video conferencing application 710, an electronic whiteboard application 720, and an interface application 730. These are application software that run on an OS (Operating System) 700. Any appropriate OS 700 may be used, and there is no requirement that it be a specific OS 700, but examples include Windows (registered trademark), Linux (registered trademark), and Android (registered trademark).

[0248] The video conference application 710 provides the functions of the transmission terminal 10 or the information processing terminal 40 described in the first embodiment. The functions of the electronic whiteboard application 720 will be described later.

[0249] 23(b) is a diagram illustrating the operation of the interface application 730. The interface application 730 is an application that provides an interface for two independent applications: a video conference application 710 and an electronic whiteboard application 720. To communicate with the interface application 730, the video conference application 710 has an interface processing unit 710a, and the electronic whiteboard application 720 has an interface processing unit 720a.

[0250] The interface application 730 communicates with the video conference application 710 via the interface processing unit 710a and with the interactive whiteboard application 720 via the interface processing unit 720a. For example, the interface application 730 detects that the video conference application 710 has received a recording start notification, or that the video conference application 710 is in the process of recording, or detects the recording management table, and notifies the interactive whiteboard application 720. This allows the interactive whiteboard application 720 to display that another location is in the process of recording.

[0251] <Example of the display screen of the electronic whiteboard 8> 24 is an example of a diagram showing the display 3 of the electronic whiteboard 8 on which an electronic whiteboard UI screen 740 generated by the electronic whiteboard application 720 is displayed. The electronic whiteboard UI screen 740 is a screen (second screen) on which images of strokes handwritten by the user are mainly displayed. The electronic whiteboard UI screen 740 also has an icon section 701, which allows the user to select the line type and color of the strokes and to draw shapes and stamps.

[0252] A video conference icon 702a is displayed in this icon section 701. When a user selects the video conference icon 702a, the interactive whiteboard 8 displays the following video conference UI screen 750 generated by the video conference application 710 on the display 3. Note that this process corresponds to switching between applications and is controlled by the OS 700.

[0253] FIG. 25 is an example diagram showing the display 3 of the interactive whiteboard 8 on which a videoconference UI screen 750 generated by the videoconference application 710 is displayed. The videoconference UI screen 750 includes an interactive whiteboard icon 703 (first screen) in addition to information equivalent to that of the destination list screen of FIG. 11. When a user selects the interactive whiteboard icon 703, the interactive whiteboard 8 displays an interactive whiteboard UI screen 740 generated by the interactive whiteboard application 720 on the display 3. Note that this process corresponds to switching between applications and is controlled by the OS. Therefore, the user can freely switch between the videoconference UI screen 750 and the interactive whiteboard UI screen 740.

[0254] It is noted that the video conference UI screen 750 and the electronic whiteboard UI screen 740 may be displayed not only alone, but also both with a partial overlap. In this case, this embodiment is effective when the other-site recording mark on the video conference UI screen 750 is hidden by the electronic whiteboard UI screen 740.

[0255] <Hardware configuration of the Electronic Whiteboard 8> 26 is a hardware configuration diagram of the electronic whiteboard 8. As shown in Fig. 26, the electronic whiteboard 8 includes a CPU 301 that controls the overall operation of the electronic whiteboard 8, a ROM 302 that stores programs such as an IPL used to drive the CPU 301, a RAM 303 that is used as a work area for the CPU 301, an SSD 304 that stores various data such as programs for the electronic whiteboard 8, a network controller 305 that controls communication with the transmission network 2, and an external storage controller 306 that controls communication with the USB memory 5.

[0256] The electronic whiteboard 8 also includes a capture device 309 that displays video information as still or moving images on the display 3 of the notebook PC 6, a GPU (Graphics Processing Unit) 308 that specializes in graphics, and a display controller 313 that controls and manages the screen display to output the output image from the GPU 308 to the display 3, etc.

[0257] Furthermore, the electronic whiteboard 8 is equipped with a sensor controller 307 that controls the processing of the contact sensor 312, and the contact sensor 312 that detects contact with the display 3 by the electronic pen 4, the user's hand H, or the like. This contact sensor 312 inputs and detects coordinates using an infrared blocking method. This coordinate input and coordinate detection method involves two light receiving and emitting devices installed at both ends of the upper side of the display 3 emitting multiple infrared rays parallel to the display 3, and receiving light that is reflected by a reflecting member installed around the display 3 and returns along the same optical path as the light emitted by the light receiving element. The contact sensor 312 outputs the ID (Identification) of the infrared rays radiated by the two light receiving and emitting devices that were blocked by an object to the sensor controller 307, and the sensor controller 307 identifies the coordinate position, which is the contact position of the object.

[0258] Furthermore, the contact sensor 312 is not limited to an infrared blocking type, and various detection means may be used, such as a capacitive touch panel that identifies the contact position by detecting changes in capacitance, a resistive film touch panel that identifies the contact position by voltage changes across two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by a contacting object coming into contact with the display unit.

[0259] The electronic whiteboard 8 also includes an electronic pen controller 310. The electronic pen controller 310 communicates with the electronic pen 4 to determine whether the tip or the butt of the pen has touched the display 3. The electronic pen controller 310 may be configured to determine whether the part of the electronic pen 4 that the user holds or other parts of the electronic pen have touched, in addition to the tip and butt of the electronic pen 4.

[0260] Furthermore, the electronic whiteboard 8 is provided with bus lines 320 such as an address bus and a data bus for electrically connecting the CPU 301, ROM 302, RAM 303, SSD 304, network controller 305, external memory controller 306, capture device 309, GPU 308, sensor controller 307, and electronic pen controller 310 as shown in FIG. 26.

[0261] <Functional configuration of the electronic whiteboard app 720> 27 to 32, the functional configuration of the electronic whiteboard 8 will be described. FIG.

[0262] 26, the electronic whiteboard 8 has an image acquisition unit 21, a coordinate detection unit 22, an automatic adjustment unit 23, a contact detection unit 24, an event allocation unit 25, an operation processing unit 26, a gesture processing unit 27, an image superimposition unit 28, and an image processing unit 9. These functions surrounded by dotted lines are functions of the video conference application 710.

[0263] Of these, the video acquisition unit 21 acquires the output video of a video output device such as a notebook PC 6 connected via a cable. When the video acquisition unit 21 receives an image signal from the video output device, it analyzes the image signal to derive image information such as the resolution of an image frame, which is the display image of the video output device formed by the image signal, and the update frequency of this image frame, and outputs the information to the image acquisition unit 31, which will be described later.

[0264] The coordinate detection unit 22 detects the coordinate position of an event (such as the user's hand H touching the display 3) that has occurred on the display 3. The coordinate detection unit 22 also detects the area of ​​the touch.

[0265] The automatic adjustment unit 23 is started when the electronic whiteboard 8 is started, and adjusts the image processing parameters of the optical sensor type sensor camera of the contact sensor 312 so that the contact sensor 312 can output an appropriate value to the coordinate detection unit 22.

[0266] The contact detection unit 24 detects an event caused by the user (such as an action of pressing (touching) the display 3 with the pen tip or the pen butt of the electronic pen 4).

[0267] The event allocation unit 25 allocates the coordinate positions of the events detected by the coordinate detection unit 22 and the detection results detected by the contact detection unit 24 to stroke drawing, UI operation, and gesture operation events. Stroke drawing, UI operation, and gesture operation will now be described with reference to Fig. 28. Fig. 28 is a configuration diagram of each image layer.

[0268] First, "stroke drawing" is performed by an event in which the user presses the electronic pen 4 on the display 3, moves the electronic pen 4 while it is pressed down, and finally releases the electronic pen 4 from the display 3. By this stroke drawing, for example, the alphabet letters "S" and "T" are drawn on the display 3. Note that this "stroke drawing" not only includes drawing an image, but also includes events such as deleting an image that has already been drawn and editing an image that has already been drawn.

[0269] A "UI operation" is an event in which the user presses a predetermined position with the electronic pen 4 or hand H when a UI image (A) shown in Fig. 28 (described later) is displayed on the display 3. This UI operation sets, for example, the color and width of the line drawn by the electronic pen 4.

[0270] A "gesture operation" is an event in which the user touches or moves the display 3 with his / her hand H when a stroke image (B) shown in Fig. 28 (to be described later) is displayed on the display 3. By performing this gesture operation, for example, the user can enlarge (or reduce) the image, change the display area, or switch pages by moving his / her hand H while touching the display 3 with his / her hand H.

[0271] 27, the operation processing unit 26 executes various operations according to the UI element on which the event occurred, based on the UI operation determined to be a UI operation by the event allocation unit 25. Examples of the UI element include a button, a list, a check box, and a text box.

[0272] The gesture processing unit 27 executes an operation corresponding to the operation determined by the event allocation unit 25 to be a gesture operation.

[0273] The video superimposing unit 28 displays the image superimposed by the display superimposing unit 36 ​​(described later) as a video on a video output device (such as the display 3). The video superimposing unit 28 also performs picture-in-picture of a video sent from another video output device (such as the camera 7) onto a video from a video output device (such as the notebook PC 6). Furthermore, the video superimposing unit 28 switches the video that has been displayed on a portion of the display 3 as a picture-in-picture video so that it is displayed on the entire display 3.

[0274] (Functional configuration of image processing unit) Next, a detailed description will be given of the functional configuration of the image processing unit 9. The image processing unit 9 performs processes such as superimposing each image layer as shown in FIG.

[0275] Therefore, the image processing unit 9 has an image acquisition unit 31, a stroke processing unit 32, a UI image generation unit 33, a background generation unit 34, a layout management unit 35, a display overlay unit 36, a page processing unit 37, a file processing unit 42, and a page data storage unit 300.

[0276] Of these, image acquisition section 31 acquires each frame as an image from the video acquired by video acquisition section 21. This image corresponds to output image (C) shown in FIG.

[0277] The stroke processing unit 32 draws an image, deletes a drawn image, or edits a drawn image based on the stroke drawing event allocated by the event allocation unit 25. The image drawn by the stroke corresponds to the stroke image (B) shown in FIG.

[0278] The UI image generating unit 33 generates a UI (user interface) image that is set in advance on the interactive whiteboard 8. This UI image corresponds to the UI image (A) shown in FIG.

[0279] The background generation unit 34 generates a background image to be displayed on the display 3. This background image corresponds to the background image (D) shown in Fig. 28. The pattern of the background image is a solid color, a grid display, or the like.

[0280] The layout management unit 35 manages layout information indicating the layout of each image output from the image acquisition unit 31, stroke processing unit 32, UI image generation unit 33, and background generation unit 34 for the display superimposition unit 36. This allows the layout management unit 35 to instruct the display superimposition unit 36 ​​at which positions in the UI image (A) and background image (D) the output image (C) and stroke image (B) should be displayed or hidden.

[0281] The display superimposition unit 36 ​​superimposes the output image (C) from the image acquisition unit 31, the stroke image (B) from the stroke processing unit 32, the UI image (A) from the UI image generation unit 33, and the background image (D) from the background generation unit 34 according to the layout specified by the layout management unit 35. As a result, as shown in Fig. 28, the layers are configured as the UI image (A), stroke image (B), output image (C), and background image (D) in the order that the user can see them even when they are superimposed.

[0282] 28, and can exclusively superimpose it on the images (A) and (B). For example, if the cable between the electronic whiteboard 8 and the video output device (such as the notebook PC 6) is unplugged while the images (A), (B), and (C) are initially displayed, the layout management unit 35 can specify that the image (C) be removed from the list of images to be superimposed, and the image (D) be displayed instead. In this case, the display superimposition unit 36 ​​also performs processes for enlarging and reducing the display, and moving the display area.

[0283] The page processing unit 37 combines the data of the stroke image (B) and the data of the output image (C) into one piece of page data and stores it in the page data storage unit 300. The page processing unit 37 also transmits the stored page data to the display superimposition unit 36 ​​via the image acquisition unit 31, so that the video superimposition unit 28 can re-display the page data on the display 3.

[0284] Furthermore, the page processing unit 37 can transmit the stored page data to the video conference application 710 via the image acquisition unit 31, thereby transmitting the page data (stroke arrangement data, media data) from the video conference application 710 to another electronic whiteboard 8 or a transmission terminal 10. Also, the page processing unit 37 can transmit the page data to the stroke processing unit 32, thereby enabling the strokes to be re-edited. Furthermore, the page processing unit 37 can delete or copy the page data.

[0285] Furthermore, when reading page data from the page data storage unit 300, the page processing unit 37 separates the combined page data into stroke array data indicating a stroke image (B) and media data indicating a background image (D). That is, the data of the output image (C) displayed on the display 3 at the time the page processing unit 37 stores the data in the page data storage unit 300 is temporarily stored in the page data storage unit 300, and when the data is subsequently read from the page data storage unit 300, it is read as media data indicating the background image (D). Then, the page processing unit 37 outputs the stroke array data indicating the stroke image (B) from the page data read from the page data storage unit 300 to the stroke processing unit 32. Furthermore, the page processing unit 37 outputs the media data indicating the background image (D) from the page data read from the page data storage unit 300 to the image acquisition unit 31.

[0286] The page data storage unit 300 stores page data as shown in Fig. 29. Fig. 12 is a conceptual diagram showing page data. Page data is data (stroke data and image data) for one page to be displayed on the display 3. Note that since there are many types of parameters included in the page data, the contents of the page data will be explained here by dividing them into Figs. 29 to 31.

[0287] After the electronic whiteboard 8 has abnormally terminated, the file processing unit 42 detects the abnormal termination and recovers the unsaved page data. For example, in the case of normal termination, the file processing unit 42 acquires the page data from the page data storage unit 300 and records it as a PDF file in the USB 5. However, in the case of abnormal termination due to a power outage or the like, the page data remains recorded in the page data storage unit 300. Therefore, when the power is turned on again, the file processing unit 42 recovers the page data by reading it from the page data storage unit 300. The file processing unit 42 also reads the PDF file from the USB memory 5 and stores each page as page data in the page data storage unit 300.

[0288] Here, page data will be described in detail. As shown in Fig. 29, the page data is stored in association with a page data ID for identifying any one page, a start time indicating the time when display of this page started, an end time indicating the time when rewriting of the page content by strokes, gestures, etc. stopped, a stroke array data ID for identifying stroke array data generated by strokes by the electronic pen 4 or the user's hand H, and a media data ID for identifying media data. The stroke array data is data for displaying a stroke image (B) shown in Fig. 11 (described later) on the display 3. The media data is data for displaying a background image (D) shown in Fig. 28 (described later) on the display 3.

[0289] With such page data, for example, when a user draws the alphabet "S" with the electronic pen 4, the letter is written in one stroke, so the single letter "S" is represented by one stroke data ID. However, when a user draws the alphabet "T" with the electronic pen 4, the letter is written in two strokes, so the single letter "T" is represented by two stroke data IDs.

[0290] The stroke array data indicates detailed information as shown in Fig. 30. Fig. 30 is a conceptual diagram showing stroke array data. As shown in Fig. 30, one piece of stroke array data is represented by a plurality of stroke data. One piece of stroke data indicates a stroke data ID for identifying this stroke data, a start time indicating the time when writing of one stroke begins, an end time indicating the time when writing of one stroke ends, the color of the stroke, the width of the stroke, and a coordinate array data ID for identifying the array of passing points of the stroke.

[0291] Furthermore, as shown in FIG. 31, this coordinate array data shows detailed information. FIG. 31 is a conceptual diagram showing the coordinate array data. As shown in FIG. 31, the coordinate array data shows information on one point (X coordinate value, Y coordinate value) on the display 3, the time difference (ms) from the disclosure time of the stroke when passing through this one point, and the pen pressure of the electronic pen 4 at this one point. That is, the collection of one point shown in FIG. 31 is shown by one coordinate array data shown in FIG. 30. For example, when the user draws the alphabet "S" with the electronic pen 4, it is a single stroke, but since a plurality of passing points are passed before finishing drawing "S", the coordinate array data shows information on these plurality of passing points.

[0292] Also, the media data among the page data shown in FIG. 29 shows detailed information as shown in FIG. 30. FIG. 32 is a conceptual diagram showing the media data. As shown in FIG. 32, the media data is associated with the media data ID in the page data shown in FIG. 29, the data type of the media data, the recording time when the page data is stored from the page processing unit 37 to the page data storage unit 300, the position (X coordinate value, Y coordinate value) and size (width, height) of the image displayed on the display 3 by the page data, and the data indicating the content of the media data. Among these, the position of the image displayed on the display 3 by the page data indicates the position of the upper left end of the image displayed by the page data when the coordinates of the upper left end of the display 3 are set as (X coordinate value, Y coordinate value) = (0, 0).

[0293] (Function Configuration of Video Conference Application) Subsequently, the function configuration of the video conference application 710 will be described in detail. The video conference application 710 controls the processing for the electronic blackboard 8 to communicate with external devices such as a transmission terminal 10, a UI providing device 20, other electronic blackboards 8, a transmission management device 50, a relay device 30, and a program providing device 90.

[0294] Therefore, the video conference application 710 has a transmission / reception unit 61, an operation input reception unit 62, a login request unit 63, a processing unit 64, an output superimposition unit 65, a display control unit 66, a destination list creation unit 67, a determination unit 68, and a storage / read processing unit 69. Furthermore, the video conference application 710 has a storage unit 6000 constructed by the SSD 304 shown in FIG.

[0295] 6. That is, the transmitting / receiving unit 61, the operation input receiving unit 62, the login request unit 63, the display control unit 66, the destination list creating unit 67, and the storage / read processing unit 69 of the video conference application 710 perform the same processes as the transmitting / receiving unit 11, the operation input receiving unit 12, the login request unit 13, the destination list display unit 16, the display control unit 17, and the storage / read processing unit 19 of the transmitting terminal 10, respectively. Also, the memory unit 6000 of the video conference application 710 plays the same role as the memory unit 1000 of the transmitting terminal 10. Therefore, the description thereof will be omitted.

[0296] On the other hand, if the display capability information (such as the screen size and resolution of the other transmitting terminal 10) received by the transmitting / receiving unit 61 indicates that the other transmitting terminal is not capable of displaying a predetermined image, the processing unit 64 processes the image data. Specifically, the processing unit 64 cuts out operation image data related to the UI image by cropping from image data in which images of multiple layers including a UI image (A) for performing operations on the electronic whiteboard 8 are superimposed, and then resizes the cropped image data to enlarge it to the image data before cropping. Note that the processing unit 64 may perform cropping after resizing to enlarge the image data.

[0297] The output superimposing unit 65 superimposes image data on image data, like the display superimposing unit 36 ​​.

[0298] If the display capability information received by the transmitting / receiving unit 61 indicates that the other transmitting terminal does not have the capability to display the specified image, the determining unit 68 determines that the image data should be processed. As a result, the processing unit 64 processes the image data.

[0299] (Interface App 730) As shown in Fig. 27, the electronic whiteboard 8 has an interface unit 730a. The interface unit 730a is a function or means realized by the CPU 301 of the electronic whiteboard 8 executing the interface application 730 and cooperating with the various hardware components shown in Fig. 26. The interface unit 730a monitors, via the interface processing unit 710a, for a fixed period of time (for example, between one second and one minute) whether or not a "recording start notification has been received" has been stored in the storage / read processing unit 69 of the videoconferencing application 710. The interface unit 730a also sends a signal to the OS 700 instructing it to select an application to be displayed in the foreground on the display 3.

[0300] The video conference application 710 also includes an interface processing unit 710a. The interface processing unit 710a is a function or means realized when the CPU 301 of the electronic whiteboard 8 executes the video conference application 710 and cooperates with the various hardware components shown in FIG. 26. When the electronic whiteboard icon 703 is pressed, the interface processing unit 710a transmits an electronic whiteboard UI screen switching signal to the interface unit 730a, indicating that the electronic whiteboard application 720 has been selected. As a result, the interface unit 730a causes the OS 700 to display the electronic whiteboard UI screen 740.

[0301] The interactive whiteboard application 720 also includes an interface processing unit 720a. The interface processing unit 720a is a function or means realized when the CPU 201 of the interactive whiteboard 8 executes the interactive whiteboard application 720 and cooperates with the various hardware components shown in FIG. 26. When the videoconferencing icon 702a is pressed, the interface processing unit 720a transmits a videoconferencing UI screen switching signal to the interface unit 730a, indicating that the videoconferencing application 710 has been selected. As a result, the interface unit 730a causes the OS 700 to display a videoconferencing UI screen 750.

[0302] <Operation procedure> Fig. 33 is an example of a sequence diagram showing an operation procedure when the video conference application 710 running on the interactive whiteboard 8 receives a recording start notification. In Fig. 33, it is assumed that a video conference has already been held through the processes in Figs. 8 and 9.

[0303] S1: A recording start notification is transmitted from the transmission terminal 10dd to the transmission management device 50. The recording start notification has been described with reference to FIG.

[0304] S2: The transmitter / receiver 51 of the transmission management device 50 transmits a recording start notification to the electronic whiteboard 8 that is holding a video conference with the transmission terminal 10dd.

[0305] S3, S4: The interface unit 730a periodically monitors whether or not a recording start notification has been received from the storage / read processing unit 69 of the video conference application 710 via the interface processing unit 710a.

[0306] S5: When a recording start notification is received, interface unit 730a determines whether this is the first time that video conference application 710 has received a recording start notification. Since storage unit 6000 of video conference application 710 also has a recording in progress management table, if a communication ID is registered when the communication ID in the recording in progress management table is zero, interface unit 730a determines that this is the first time that video conference application 710 has received a recording start notification. This eliminates the need to notify video conference application 710 every time a recording start notification is received.

[0307] S6, S7: When the video conference application 710 determines that it has received a recording start notification for the first time, the interface unit 730a makes a UI change request to the UI image generation unit 33 via the interface processing unit 720a. The UI change request is a request to display the operating status of the transmitting terminal 10dd on the electronic whiteboard UI screen 740. Specifically, it displays that the transmitting terminal 10dd is recording.

[0308] S8: As a result, the UI image generating unit 33 of the interactive whiteboard application 720 displays the interactive whiteboard UI screen 740 indicating that the transmission terminal 10dd is recording. An example of this interactive whiteboard UI screen 740 will be described with reference to FIG.

[0309] At this point, the interface unit 730a does not need to send a videoconference UI screen switching signal to the OS 700. First, when the electronic whiteboard UI screen 740 is displayed on the display 3 of the electronic whiteboard 8, a message indicating that the transmission terminal 10dd is recording is displayed in step S8, allowing the user to notice that recording is being performed at another location. When the videoconference UI screen 750 is displayed on the display 3 of the electronic whiteboard 8, the recording location mark 71 or the other location recording mark 72 described in the first embodiment is displayed on the full-screen display screen 501 or the information display screen 601, allowing the user to notice that recording is being performed at another location.

[0310] Similarly, the interface unit 730a can detect that the video conference application 710 has received a recording end notification. In this case, the interface unit 730a makes a UI change request to the UI image generation unit 33 of the electronic whiteboard application 720. This UI change request is a request to erase the information that the transmitting terminal 10dd is recording.

[0311] FIG. 34 is an example of an interactive whiteboard UI screen 740 that displays information indicating that another location is recording. In FIG. 34, the text 761 "Recording Conference in Progress" is displayed in the ticker section 760. The ticker section 760 is an area on the screen that displays information unrelated to the content, such as an image. Generally, this area is created by shrinking the content portion while maintaining the aspect ratio of the content portion. This allows the display of information indicating that another location is recording without overlapping with the content.

[0312] However, the interactive whiteboard UI screen 740 does not need to be particularly reduced in size in FIG. 34, and may be displayed superimposed on the four corners or center of the content (handwritten strokes, etc.) of the interactive whiteboard UI screen 740, for example.

[0313] 34, the UI image generating unit 33 may display the fact that recording is in progress by flashing the video conference icon 702a, changing the color, etc. Also, instead of displaying the text 761, an icon indicating that recording is in progress may be displayed.

[0314] <Change display order> Next, a procedure for changing the display order of the electronic whiteboard UI screen 740 and the video conference UI screen 750 will be described with reference to Fig. 35. Fig. 35 is an example of a sequence diagram showing a procedure by which the interface unit 730a changes the display order of the electronic whiteboard UI screen 740 and the video conference UI screen 750. Fig. 35 is executed while a video conference is being held. The description will be made assuming that the video conference UI screen 750 is displayed on the display 3 of the electronic whiteboard 8.

[0315] S1: The user presses the electronic whiteboard icon 703 on the video conference UI screen 750. The operation input receiving unit 62 receives this operation and notifies the interface processing unit 710a.

[0316] S2: Upon receiving this notification, the interface processing unit 710a transmits an electronic whiteboard UI screen switching signal to the interface unit 730a.

[0317] S3: The interface unit 730a transmits an electronic whiteboard UI screen switching signal to the OS 700.

[0318] S4: The OS 700 displays the electronic whiteboard UI screen 740 generated by the video superimposition unit 28 of the electronic whiteboard application 720 in the foreground on the display 3. The OS 700 also activates the electronic whiteboard UI screen 740. That is, the electronic whiteboard application 720 that created the electronic whiteboard UI screen 740 becomes the target of user operation. The OS 700 also assigns a slightly higher priority to the electronic whiteboard application 720 than to other applications. As a result, the electronic whiteboard UI screen 740 is displayed in the foreground on the display 3 while the videoconferencing application 710 remains running.

[0319] In addition, when the interactive whiteboard application 720 is displayed in the foreground, the video conference application 710 can be displayed in the foreground by performing the reverse procedure of FIG.

[0320] In this way, the user can switch between the electronic whiteboard application 720 and the video conference application 710 at will.

[0321] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0322] For example, the relay device 30, the transmission management device 50, and the program providing device 90 may be implemented by a single computer, rather than being independent of each other. Furthermore, the means and the storage units 1000, 2000, and 5000 of the transmission terminal 10, the transmission management device 50, and the program providing device 90 may be arbitrarily assigned to multiple computers. In this case, the UI providing device 20 may be included as a function of any of the computers.

[0323] Furthermore, in the above embodiment, a video conference system has been described as an example of the transmission system 1, but the present invention is not limited to this and may be a telephone system such as an IP (Internet Protocol) telephone or an Internet telephone.

[0324] Furthermore, each transmission terminal 10 may communicate without going through the transmission management device 50. WebRTC is known as an example of such a communication protocol.

[0325] The transmission system 1 may also be a system including a car navigation device mounted on a mobile object. A PC for screen sharing may also be connected to the transmission terminal 10.

[0326] In the above embodiment, image data and audio data are described as examples of content data, but the present invention is not limited to these and may also be applied to tactile data. In this case, the sensation of a user touching one transmission terminal is transmitted to the other transmission terminal.

[0327] Furthermore, the content data may be smell data. In this case, the smell (odor) from one transmitting terminal is transmitted to the other transmitting terminal. The content data may be at least one of image data, audio data, tactile data, and smell data.

[0328] In addition, in the above embodiment, the transmission system 1 is used to hold a video conference, but this is not limited to this, and it may also be used for meetings, general conversations between family members or friends, or for presenting information in one direction.

[0329] In addition, in the above embodiment, participants were mainly notified that image data was being recorded, but participants may also be notified that audio data is being recorded when only audio data is being sent and received.

[0330] The transmitting / receiving unit 11 is an example of a receiving means, the base list display unit 18b is an example of a recording terminal notifying means, the recording in progress management table 1000b is an example of an identification information storing means, the full-screen display screen 501 is an example of a first screen, and the information display screen 601 is an example of a second screen. The transmission management device 50 is an example of an external device, the destination list display unit 16 is an example of a destination list display means, and the display 120 is an example of a display device. The display control unit 17 is an example of a display control means, the operation input accepting unit 12 is an example of a selection accepting means, and the auxiliary area display unit 18a is an example of a recording in progress notifying means. The destination list management DB 5003 is an example of a destination list storing means, the terminal management DB 5002 is an example of a terminal status storing means, the image recording unit 41 is an example of a recording means, and the transmitting / receiving unit 51 is an example of a transmitting means. The transmission terminal 10 is an example of a first information processing device (or a second information processing device), and the transmission management device 50 is an example of a second information processing device. The display control unit 66 is an example of a first display control means. The UI image generating unit 33 and the video superimposing unit 28 are an example of a second display control unit, and the interface unit 703a is an example of a notification unit. [Explanation of symbols]

[0331] 1 Transmission System 10 Transmission terminal 12 Operation input reception section 17 Display control unit 18a Auxiliary area display section 18b Location list display section 50 Transmission management device 52 Terminal authentication unit 53 State Management Unit 54 Terminal Extraction Unit 55 Terminal status acquisition unit 120 Display [Prior art documents] [Patent documents]

[0332] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-216984

Claims

1. An information processing device that transmits at least one of image data and audio data to one or more other information processing devices connected via a network, a receiving means for receiving a recording notification indicating that the image data or audio data transmitted by the information processing device will be recorded by the other information processing device together with identification information of the other information processing device; a recording terminal notifying means for notifying the other information processing device that the other information processing device that transmitted the recording notification is currently recording image data or audio data, based on the identification information received by the receiving means, in association with the other information processing device; An information processing device having the above.

2. an identification information storage means for storing the identification information received by the receiving means; The information processing device according to claim 1, wherein the recording terminal notification means reads out all of the identification information from the identification information storage means and displays on a display device for each of the other information processing devices that the other information processing device of the identification information is recording image data or audio data.

3. When at least one of the one or more other information processing devices communicating with the other information processing device has transmitted the recording notification, 3. The information processing apparatus according to claim 2, wherein said recording terminal notifying means always displays on a display device that said other information processing apparatus is currently recording.

4. a display control means for switching between a first screen for displaying image data transmitted from the other information processing device on a display device and a second screen for displaying information relating to the other information processing device on a display device; a recording notification means for displaying, on the first screen, recording notification information indicating that the other information processing device is recording image data or audio data while the first screen is being displayed, when at least one of the identification information is stored in the identification information storage means; 4. The information processing apparatus according to claim 3, wherein the recording terminal notifying means displays on the second screen information about the other information processing apparatus as well as information that the other information processing apparatus is currently recording.

5. When no identification information is stored in the identification information storage means and the receiving means receives the recording notification while the first screen is being displayed, the recording notification means starts displaying the recording notification information on the first screen, 5. The information processing device according to claim 4, wherein when the receiving means receives the recording notification while the second screen is being displayed, the recording terminal notification means starts displaying a message indicating that the other information processing device is recording image data or audio data.

6. When the receiving means receives a recording end notification indicating that the other information processing device has ended recording of the image data or audio data transmitted by the information processing device, together with identification information of the other information processing device, the recording in progress notification means deletes the identification information of the other information processing device that has transmitted the recording end notification from the identification information storage means, and when no identification information is stored in the identification information storage means, hides the recording notification information on the first screen; 6. The information processing apparatus according to claim 4, wherein the recording terminal notifying means does not display on the second screen any information that the other information processing apparatus is currently recording.

7. When the receiving means receives, from an external device that monitors whether the other information processing device is alive or dead, information indicating a change in the operation state indicating that communication has been disconnected together with identification information of the other information processing device, the recording notification means deletes the identification information of the other information processing device transmitted by the external device from the identification information storage means, and when no identification information is stored in the identification information storage means, hides the recording notification information on the first screen; 7. The information processing apparatus according to claim 6, wherein the recording terminal notifying means does not display on the second screen any information that the other information processing apparatus is currently recording.

8. an external device that has a destination list of the other information processing devices with which the information processing device can start communication and transfers the recording notification transmitted by the other information processing device and the identification information of the other information processing device to the information processing device; the receiving means receives information about an operating state including whether or not recording is being performed for each of the other information processing devices in the destination list; 8. The information processing apparatus according to claim 1, further comprising a destination list display means for displaying on a display device whether the other information processing apparatus in the destination list is currently recording or not, based on the information regarding the operating state.

9. a selection receiving means for receiving a selection of the other information processing device that is a communication destination; 9. The information processing apparatus according to claim 8, wherein when the selection receiving means receives the selection of the other information processing apparatus that is being recorded, the destination list display means displays on the display device a message indicating that communication will be performed with the other information processing apparatus that is being recorded.

10. a first display control means for displaying the image data received from the other information processing device on a first screen of a display device; a second display control means for displaying a second screen on a display device, the second screen being different from the first screen on which the image data is displayed; An information processing device as described in any one of claims 1 to 9, further comprising: a notification means for causing the second display control means to display on the second screen, when the receiving means receives the recording notification, a message indicating that the information displayed on the second screen is being recorded by the other information processing device.

11. 11. The information processing apparatus according to claim 10, wherein the notification means displays either the first screen or the second screen on the forefront of a display device.

12. A transmission system having a plurality of information processing devices that transmit at least one of image data and audio data via a network, The first information processing device a recording means for recording image data or audio data transmitted by the second information processing device; a transmitting means for transmitting a recording notification that the recording means has started recording; The second information processing device a receiving means for receiving the recording notification together with identification information of the first information processing device; a recording terminal notifying means for notifying the first information processing device that the first information processing device that transmitted the recording notification is currently recording image data or audio data, based on the identification information received by the receiving means, in association with the first information processing device; A transmission system having:

13. A transmission system including a plurality of first information processing devices that transmit at least one of image data and audio data via a network, and a second information processing device that manages communication between the first information processing devices, The second information processing device a destination list storage means for storing a destination list of other information processing devices with which the first information processing device can start communication; a terminal status storage means for storing an operating status in which whether or not the other information processing device is recording image data or audio data transmitted by the first information processing device is registered; a transmitting means for transmitting the operating status to the first information processing device, The first information processing device a receiving means for receiving the operating status of each of the other information processing devices in the destination list; a destination list display means for displaying on a display device whether the other information processing device in the destination list is recording image data or audio data based on the operating status; A transmission system having:

14. The second information processing device A transmission system as described in claim 13, wherein the operating status of all other information processing devices and the first information processing device that are communicating with the other information processing device that is recording image data or audio data is registered as indicating that image data or audio data is being recorded.

15. An information processing device that transmits at least one of image data and audio data to one or more other information processing devices connected via a network, a receiving means for receiving a recording notification indicating that the image data or audio data transmitted by the information processing device will be recorded by the other information processing device together with identification information of the other information processing device; a recording terminal notifying means for notifying the other information processing device that the other information processing device that transmitted the recording notification is currently recording image data or audio data, based on the identification information received by the receiving means, in association with the other information processing device; A program to function as a

Images