Control system, control method, and computer program

Abstract

To realize screen display having higher convenience for a user in a system capable of providing a plurality of moving images.SOLUTION: In a virtual conference room where a user and other users can make conversation via a network, there is provided a control unit which displays a partial region where images provided from user terminals of the other users and a reduced partial region for each of one or more users attending to the same conference room as the user. The reduced partial region is a region relatively smaller than the partial region. The control unit assigns and displays a user who is transmitting a moving image to the partial region, and assigns and displays a user who is not transmitting the moving image to the reduced partial region.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique capable of displaying a plurality of moving images provided online. [Background technology]

[0002] Conventionally, systems for conducting conversations and conferences via a network have been proposed. In these systems, facial images of participants captured by a camera and images of windows shared by participants are generally displayed in partial areas provided for each user. For example, Patent Document 1 describes a method of extracting an image of a captured facial area of ​​a user and displaying it with a blurred background. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-213013 A Summary of the Invention [Problem to be solved by the invention]

[0004] Conventionally, there are systems in which, when multiple members participate, a partial area is provided for each of one or more participating members, and faces and moving images are displayed. There are also systems in which the screen layout of the partial area for each member can be selected from multiple predefined screen layouts. For example, there is a screen layout in which a partial area is assigned to all participating members, and an image of each member is displayed in the partial area. As another example, there is a screen layout in which the partial area of ​​a representative member (for example, the member who is speaking at that time) is set larger than the partial areas of other members. As another example, there is a screen layout in which a partial area for displaying an image of a shared window or desktop is set larger than the partial areas of other members. However, sufficient convenience has not always been realized. Such a problem is not necessarily limited to systems for conversations and meetings, but is a common problem in systems that can display multiple moving images provided by other users or other systems.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique that enables a screen display that is more convenient for users to realize in a system capable of providing a plurality of moving images. [Means for solving the problem]

[0006] One aspect of the present invention is a control system that, in a virtual conference room for a user to converse with other users over a network, includes a control unit that displays, for each one or more other users participating in the same conference room as the user, a partial area and a reduced partial area in which images provided from the user terminals of the other users can be displayed on the screen of the user's user terminal, the reduced partial area being an area relatively smaller than the partial area, and the control unit assigns users who are sending moving images to the partial areas and assigns users who are not sending moving images to the reduced partial areas for display.

[0007] One aspect of the present invention is the above-mentioned control system, wherein the control unit sets target users to be assigned to the partial area for users who are participating in the same conference room as the user, and assigns and displays users who are not set as target users to the reduced partial area, even if they are users sending moving images.

[0008] One aspect of the present invention is the above-mentioned control system, wherein the control unit removes the user from the target users when the user performs an operation indicating removal from the target users.

[0009] One aspect of the present invention is a control method in which a computer displays, in a virtual conference room for a user and other users to converse with each other over a network, a partial area and a reduced partial area in which images provided from the user terminals of one or more other users participating in the same conference room as the user, on the screen of the user's user terminal, the reduced partial area being an area relatively smaller than the partial area, and in the control step, users who are sending moving images are assigned to the partial area and users who are not sending moving images are assigned to the reduced partial area for display.

[0010] One aspect of the present invention is a computer program for causing a computer to function as a terminal device that, in a virtual conference room for a user to converse with other users over a network, includes a control unit that displays, for each one or more other users participating in the same conference room as the user, a partial area and a reduced partial area in which images provided from the user terminal of the other users can be displayed, the reduced partial area being an area relatively smaller than the partial area, and the control unit assigns users who are sending moving images to the partial areas and assigns users who are not sending moving images to the reduced partial areas for displaying them. Effect of the Invention

[0011] According to the present invention, in a system capable of providing a plurality of moving images, it is possible to realize a screen display that is more convenient for the user. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic block diagram showing a system configuration of a conversation control system 100 of the present invention. [Diagram 2] 2 is a schematic block diagram showing a specific example of the functional configuration of a user terminal 10. FIG. [Diagram 3] 2 is a schematic block diagram showing a specific example of the functional configuration of the conversation control device 20. FIG. [Figure 4] 2A and 2B are diagrams showing specific examples of images displayed on the display unit 13 of the user terminal 10. [Diagram 5] 2A and 2B are diagrams showing specific examples of images displayed on the display unit 13 of the user terminal 10. [Figure 6] 2A and 2B are diagrams showing specific examples of images displayed on the display unit 13 of the user terminal 10. [Figure 7] 2A and 2B are diagrams showing specific examples of images displayed on the display unit 13 of the user terminal 10. [Figure 8] 2A and 2B are diagrams showing specific examples of images displayed on the display unit 13 of the user terminal 10. [Figure 9] 2A and 2B are diagrams showing specific examples of images displayed on the display unit 13 of the user terminal 10. [Figure 10] 2A and 2B are diagrams showing specific examples of images displayed on the display unit 13 of the user terminal 10. [Figure 11] 2A and 2B are diagrams showing specific examples of images displayed on the display unit 13 of the user terminal 10. [Figure 12] 2A and 2B are diagrams showing specific examples of images displayed on the display unit 13 of the user terminal 10. [Figure 13] 13 is a diagram showing a specific example of a state in which screen sharing is being performed in a reduced partial area 51. FIG. [Figure 14] 13 is a diagram showing a specific example of a state in which screen sharing is being performed in a reduced partial area 51 and a partial area 52. FIG. [Figure 15] FIG. 13 is a diagram showing a display example in an automatic mode. [Figure 16] FIG. 13 is a diagram showing a display example in an automatic mode. [Figure 17] FIG. 13 is a diagram showing a display example in an automatic mode. [Figure 18] FIG. 13 is a diagram showing a display example in an automatic mode. [Figure 19] FIG. 13 is a diagram showing a display example in a focus mode. [Figure 20] FIG. 13 is a diagram showing a display example in a focus mode. [Figure 21] FIG. 13 is a diagram showing a display example in a focus mode. [Figure 22] FIG. 9 is a diagram illustrating an outline of a hardware configuration example of an information processing device 900 applied to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, specific configuration examples of the present invention will be described with reference to the drawings. FIG. 1 is a schematic block diagram showing a system configuration of a conversation control system 100 of the present invention. The conversation control system 100 is a specific example of a system capable of providing a plurality of moving images. The conversation control system 100 is a system in which users operating user terminals 10 can have a conversation with each other via a network 90. ​​The conversation control system 100 includes a plurality of user terminals 10 and a conversation control device 20. The plurality of user terminals 10 and the conversation control device 20 are communicatively connected via the network 90. ​​The network 90 may be a network using wireless communication or a network using wired communication. The network 90 may be configured by combining a plurality of networks.

[0014] 2 is a schematic block diagram showing a specific example of the functional configuration of the user terminal 10. The user terminal 10 is configured using information devices such as a smartphone, a tablet, a personal computer, a portable game machine, a stationary game machine, a dedicated device, etc. The user terminal 10 includes a communication unit 11, an operation unit 12, a display unit 13, an audio input unit 14, an audio output unit 15, an imaging unit 16, a storage unit 17, and a control unit 18.

[0015] The communication unit 11 is a communication device. The communication unit 11 may be configured as, for example, a network interface. The communication unit 11 performs data communication with other devices via a network 90 in response to control by the control unit 18. The communication unit 11 may be a device that performs wireless communication or a device that performs wired communication.

[0016] The operation unit 12 is configured using existing input devices such as a keyboard, a pointing device (mouse, tablet, etc.), buttons, and a touch panel. The operation unit 12 is operated by the user when inputting the user's instructions to the user terminal 10. The operation unit 12 may be an interface for connecting the input device to the user terminal 10. In this case, the operation unit 12 inputs an input signal generated in the input device in response to the user's input to the user terminal 10. The operation unit 12 may be configured using a microphone and a voice recognition device. In this case, the operation unit 12 recognizes the words spoken by the user and inputs character string information of the recognition result to the user terminal 10. In this case, the voice recognition process may be executed by the control unit 18. In this case, the microphone constituting the operation unit 12 may be configured integrally with the voice input unit 14 described later. The operation unit 12 may be configured in any manner as long as it is configured to input the user's instructions to the user terminal 10.

[0017] The display unit 13 is an image display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 13 displays image data used in the conversation control system 100. The display unit 13 may be an interface for connecting an image display device to the user terminal 10. In this case, the display unit 13 generates a video signal for displaying the image data and outputs the video signal to the image display device connected to itself.

[0018] The voice input unit 14 is configured using a microphone. The voice input unit 14 may be configured as a microphone itself, or may be configured as an interface for connecting a microphone as an external device to the user terminal 10. The microphone acquires the speech of a user having a conversation. The voice input unit 14 outputs data of the voice acquired by the microphone to the control unit 18.

[0019] The audio output unit 15 is configured using an audio output device such as a speaker, headphones, or earphones. The audio output unit 15 may be configured as the audio output device itself, or may be configured as an interface for connecting the audio output device to the user terminal 10 as an external device. It is desirable for the audio output device to output audio so that the user having a conversation can hear the audio. The audio output unit 15 outputs audio corresponding to an audio signal output by the control unit 18.

[0020] The imaging unit 16 is configured using an imaging device such as a camera. The imaging unit 16 may be an interface for connecting an imaging device such as a camera to the user terminal 10. The imaging unit 16 is preferably installed in a position where it can capture an image of the user's face. In this case, the imaging unit 16 captures an image of the user's face and generates still image or video data (hereinafter referred to as "captured image data"). The imaging unit 16 outputs the generated captured image data to the control unit 18. The captured image data is one aspect of a provided video. The provided video is a video provided from the user terminal 10 or an information device to another user terminal 10. Specific examples of the provided video include a video (captured image data) captured by the imaging unit 16 of the user terminal 10 and a video showing a screen shared in the user terminal 10 (hereinafter referred to as a "shared screen"). The shared screen is, for example, a window selected by the user, a window displaying a selected file, a selected desktop, or the like. In the video showing the shared screen, operations performed by the user on the screen may be reflected at any time. For example, when a user operates the position of a cursor, the display position of the cursor changes on the shared screen in response to the operation.

[0021] The storage unit 17 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 17 stores data used by the control unit 18. The storage unit 17 may function as, for example, a user information storage unit 171. The user information storage unit 171 stores information about the user who operates the user terminal 10 (hereinafter referred to as "user information"). The user information may include, for example, the user's handle name, the user's icon data, and the user's attribute information. The handle name is the name (nickname or real name) that the user uses when having a conversation in the conversation control system 100. The icon data is image data that the user uses when having a conversation in the conversation control system 100. The attribute information may include, for example, information about the user's age, sex, and affiliation.

[0022] The control unit 18 is configured using a processor such as a CPU (Central Processing Unit) and a memory. The control unit 18 functions as a display control unit 181, a conference control unit 182, and a conversation control unit 183 by the processor executing a program. All or part of the functions of the control unit 18 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a semiconductor storage device (for example, an SSD: Solid State Drive), or a storage device such as a hard disk or a semiconductor storage device built into a computer system. The above program may be transmitted via an electric communication line.

[0023] The display control unit 181 receives display information from the conversation control device 20 via the communication unit 11. The display control unit 181 generates an image signal based on the acquired display information and displays it on the display unit 13. At this time, the display control unit 181 performs display control within the display area of ​​the application in which the application of the conversation control system 100 is running. For example, if the application of the conversation control system 100 is a web browser, display control is performed within the window of the web browser. If the web browser is displayed full screen, display control is performed on the entire display screen of the display unit 13.

[0024] The display information may be, for example, image data indicating the image itself to be displayed. In this case, the entity that generates the image data (image data generation unit) is the display information generation unit 233 of the conversation control device 20. The display information may be, for example, data indicating information required to generate the image to be displayed (for example, information regarding the conference room (participants, number of participants, etc.), the number of divided areas to be displayed, the shape of the image area to be displayed in each divided area, the position, size and display mode of the conference room, and information indicating the position and icon image of the user). In this case, the display control unit 181 generates image data to be displayed on the display unit 13 based on the display information. In this case, the entity that generates the image data (image data generation unit) is the display control unit 181.

[0025] The conference control unit 182 performs control related to a conference virtually established in the conversation control device 20. For example, when a user instructs to set up a new conference room by operating the operation unit 12, the conference control unit 182 performs processing for setting up a new conference room. When the user of the own device (user terminal 10) has set up the captured image data to be transmitted to another user terminal 10, the conference control unit 182 transmits the captured image data captured by the imaging unit 16 to the other device. The other device as the transmission destination may be, for example, the conversation control device 20. In this case, the captured image data is transmitted to the other user terminal 10 connected to the same conference room as the own device (user terminal 10) via the conversation control device 20. The other device as the transmission destination may be the other user terminal 10 connected to the same conference room as the own device (user terminal 10). When the user of the own device (user terminal 10) has set up the transmission of the shared screen to the other user terminal 10, the conference control unit 182 transmits data of a moving image showing the shared screen in the own device to the other device. The other device as the transmission destination may be, for example, the conversation control device 20. In this case, the video data of the shared screen is transmitted to the other user terminals 10 connected to the same conference room as the user terminal 10 via the conversation control device 20. The other device to which the data is transmitted may be the other user terminal 10 connected to the same conference room as the user terminal 10.

[0026] The conversation control unit 183 controls a conversation between multiple users participating in the conversation control system 100. A conversation between multiple users may be, for example, a conversation between multiple users participating in a conference room (hereinafter referred to as a "conference room conversation"). When the user of the own device (user terminal 10) participates in a conversation, the conversation control unit 183 controls a conversation between multiple users including the user of the own device. In this case, the conversation control unit 183 transmits, for example, voice data input from the voice input unit 14 to the conversation control device 20 via the communication unit 11. When the conversation control unit 183 receives voice data from the conversation control device 20, it outputs the received voice data from the voice output unit 15. When the user of the own device (user terminal 10) does not participate in the conversation, the conversation control unit 183 controls a conversation between multiple users not including the user of the own device. In this case, the conversation control unit 183 receives voice data of a conversation between other users and outputs it from the voice output unit 15. The voice data output from the voice output unit 15 may be received from another user terminal 10 via the conversation control device 20, or may be received directly from another user terminal 10 connected in accordance with the control of the conversation control device 20.

[0027] 3 is a schematic block diagram showing a specific example of a functional configuration of the conversation control device 20. The conversation control device 20 is configured using an information processing device such as a personal computer or a server device. The conversation control device 20 includes a communication unit 21, a storage unit 22, and a control unit 23.

[0028] The communication unit 21 is a communication device. The communication unit 21 may be configured as, for example, a network interface. The communication unit 21 performs data communication with other devices via the network 90 in accordance with the control of the control unit 23. The communication unit 21 may be a device that performs wireless communication or a device that performs wired communication.

[0029] The storage unit 22 is configured using a storage device such as a magnetic hard disk device, a semiconductor storage device, etc. The storage unit 22 stores data used by the control unit 23. The storage unit 22 may function as a conference room information storage unit 221, for example.

[0030] The conference room information storage unit 221 stores information about conference rooms (hereinafter referred to as "conference room information"). A conference room is a virtual room set up in the conversation control system 100 for users to have conversations. The conference room information may include, for example, information indicating the name set for the conference room. The conference room information may further include information indicating the users currently in the conference room.

[0031] The control unit 23 is configured using a processor such as a CPU and a memory. The control unit 23 functions as a user control unit 231, a conference room control unit 232, a display information generating unit 233, and a conversation control unit 234 by the processor executing a program. All or part of the functions of the control unit 23 may be realized using hardware such as an ASIC, a PLD, or an FPGA. The above program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a semiconductor storage device (e.g., SSD), or a storage device such as a hard disk or a semiconductor storage device built into a computer system. The above program may be transmitted via an electric communication line.

[0032] The user control unit 231 performs control processing related to users. For example, the user control unit 231 may perform processing for logging in (for example, authentication processing) for a user who accesses the conversation control device 20.

[0033] The conference room control unit 232 performs control processing related to the conference room. For example, when the conference room control unit 232 receives an instruction from the user terminal 10 to set up a new conference room, the conference room control unit 232 may generate conference room information based on the received information and register it in the conference room information storage unit 221. When a predetermined operation is performed by a user to participate in a conference room, the conference room control unit 232 performs processing to allow the user to participate in the conference room if a predetermined condition is satisfied. For example, the conference room control unit 232 updates the conference room information storage unit 221 to register that a new user has participated in the conference room.

[0034] The display information generating unit 233 generates information (display information) required to generate an image to be displayed on the user terminal 10. The display information may include, for example, the number of divided areas, the shape of the image area displayed in each divided area, information about the currently installed conference room, and information about the user of each user terminal 10 accessing the conversation control device 20. The display information generating unit 233 transmits the generated display information to the user terminal 10.

[0035] The conversation control unit 234 receives voice data from the user terminal 10. The conversation control unit 234 generates voice data to be output to each user terminal 10 (hereinafter referred to as "conversation voice data") and transmits the conversation voice data to each user terminal 10. The conversation control unit 234 may generate conversation voice data based on, for example, what kind of conversation the user of each user terminal 10 is participating in. For example, the conversation control unit 234 may transmit, to each user's user terminal 10, conversation voice data including voices received from the user terminals 10 of all users participating in the conference room in which that user is present.

[0036] Next, the relationship between the images displayed on the user terminal 10 and the conversation voice data output when each image is displayed will be described.

[0037] FIG. 4 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In FIG. 4, a conference room area image 30, a conversation area 31, a listening area 32, an icon image 41, and an operation area image 501 are displayed on the display unit 13. An image of a virtual space in which a conference room can be set up is displayed on the display unit 13. The virtual space is mainly composed of a conference room area and an area outside the conference room. The conference room area is displayed as a conference room area image 30. The area outside the conference room (hereinafter referred to as the "outdoor area") is the area represented by the symbol 91, and is an area in which icon images 41 of users who are not in the conference room are displayed.

[0038] The conference room area image 30 includes a conversation area 31 and a listening area 32. A user who has an icon image 41 displayed in the conversation area 31 can converse with other users who have an icon image 41 displayed in the conversation area 31 of the same conference room. That is, the voices of the users who have an icon image 41 displayed in the conversation area 31 are transmitted and received between each other. In the following description, entering a conference room means that the icon image 41 is displayed in the conversation area 31. That is, entering a conference room means that a user is in a state in which he or she can converse with other users in the conference room.

[0039] A user whose icon image 41 is displayed in the listening area 32 (hereinafter referred to as a "listening user") can listen to the conversation of a user whose icon image 41 is displayed in the conversation area 31 of the conference room (a user who has entered the conference room). However, the listening user cannot speak in the conference room. In other words, the voice uttered by the listening user is not output in the user terminals 10 of other users who have entered the conference room, or in the user terminals 10 of other listening users in the conference room.

[0040] The conference room area image 30 is displayed as a geometric image showing an area of ​​a predetermined size. In the example of this embodiment, the conference room area image 30 is displayed as a circular image. The name of the conference room 311 is displayed near the center of the conference room area image 30. In the example of FIG. 4, the name of the conference room is "Chat." By displaying the name of the conference room, even a user located in the outdoor area 91 can roughly guess what kind of conversation is taking place in the conference room without entering the conference room.

[0041] An entry button 312 and a details button 313 may be displayed inside the conference room area image 30. The entry button 312 is a button for requesting entry into the conference room. In response to the entry button 312 being operated by a user located in the outdoor area 91, the conference room control unit 232 determines that the user is requesting entry into the conference room. In response to the request, the conference room control unit 232 determines whether or not to permit entry, and controls the user's entry in accordance with the determination result.

[0042] The details button 313 is a button for displaying detailed information about the conference room. In response to the operation of the details button 313, a details display area is displayed on the screen. In the details display area, detailed information about the conference room, and buttons and input frames for performing operations on the conference room are displayed. For example, the name of the conference room, a button for changing the name of the conference room, an input frame for entering the changed name of the conference room, etc. may be displayed.

[0043] The icon image 41 is an image that indicates the virtual position of the user and information about the user. The icon image 41 may be displayed according to icon data included in the user information, for example.

[0044] 5 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of FIG. 5, icon images 41 (41_1 and 41_2) of two users and a conference room area image 30 of one conference room are displayed. No user has yet participated in the conference room indicated by the conference room area image 30. The two users have not participated in any conference room. Therefore, the icon image 41_1 and the icon image 41_2 are displayed in the outside area 91, not inside the conference room area image 30.

[0045] A user who wishes to participate in a conference room can do so (enter) the conference room by performing a specific operation. For example, a user can join the conference room by operating the entry button 312 in the conference room area image 30 in which the user wishes to participate. Note that the operation for entering the room does not need to be limited to the above. If an entry restriction is set for a conference room, only users who are permitted to enter the conference room according to the entry restriction can enter the conference room.

[0046] Fig. 6 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of Fig. 6, one user is participating in a conference room with the theme of chatting. Therefore, an icon image 41 (41_2) of one user is displayed inside the conference room area image 30 showing the conference room with the theme of chatting. At this point, there are no other users participating in the same conference room as the user of icon image 41_2. Therefore, the user of icon image 41_2 cannot talk with other users in the conference room with the theme of chatting.

[0047] When a user of the icon image 41_1 in the outdoor area 91 performs an operation to participate in a conference room with a chat theme, the users participating in the conference room are notified that there is a user who wishes to participate. For example, a predetermined sound (e.g., a door knocking sound) may be output from the user terminal 10 of each user. The users to be notified may be all users participating in the conference room, or only a predetermined portion of users (e.g., the user who is the administrator of the conference room).

[0048] If a predetermined condition is met after the notification, the user of the icon image 41_1 can participate in the conference room with the theme of chatting. The predetermined condition may be, for example, simply that a predetermined time (e.g., 1 second, 3 seconds, 5 seconds, etc.) has passed, or that participation has been permitted by a predetermined user (e.g., the administrator or participant of the conference room), or that a participation condition preset for the conference room has been met. Also, a predetermined condition may not be set. In that case, all users who wish to participate can participate in the conference room (e.g., a conference room with the theme of chatting).

[0049] FIG. 7 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of FIG. 7, two users are participating in a conference room with the theme of chatting. Therefore, icon images 41 (41_1 and 41_2) of the two users are displayed inside the conference room area image 30 showing the conference room with the theme of chatting. At this point, the user of the icon image 41_1 and the user of the icon image 41_2 are participating in the same conference room. Therefore, the user of the icon image 41_1 and the user of the icon image 41_2 can have a conversation in the conference room with the theme of chatting. In other words, users participating in the same conference room can have a conversation using the user terminal 10.

[0050] More specifically, it is as follows. The user terminal 10 of a user participating in a conference room receives and outputs voice data spoken by other users participating in the same conference room. In addition, the user terminal 10 of a user participating in a conference room transmits the voice data spoken by that user to the conversation control device 20. This transmission causes the voice data to be output in the user terminals 10 of the other users participating in the same conference room. When a user of icon image 41_3 in the outdoor area 91 performs an operation to participate in a conference room with a chat theme, after a predetermined condition is met, the user of icon image 41_3 can participate in the conference room with a chat theme.

[0051] FIG. 8 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of FIG. 8, three users participate in a conference room with the theme of chatting. Therefore, icon images 41 (41_1, 41_2, and 41_3) of the three users are displayed inside the conference room area image 30 showing the conference room with the theme of chatting. At this point, the user of the icon image 41_1, the user of the icon image 41_2, and the user of the icon image 41_3 participate in the same conference room. Therefore, the user of the icon image 41_1, the user of the icon image 41_2, and the user of the icon image 41_3 can have a conversation in the conference room with the theme of chatting. The same applies when four or more users participate in the same conference room.

[0052] 9 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of FIG. 9, six users are participating in a conference room with the theme of chatting. In the example of FIG. 9, the user terminal 10 is operating in a video conversation mode. In the video conversation mode, a reduced partial area 51 is displayed for each user participating in the same conference room (the conference room with the theme of chatting in FIG. 9). The reduced partial area 51 is an area capable of displaying text, still images, and moving images (provided moving images) provided from the user terminal 10 of the user associated with that reduced partial area 51.

[0053] Types of video images other than those described above may be displayed as provided video images in the reduced partial areas 51. In the example of Fig. 9, provided video images including the faces of the users captured by the imaging unit 16 of the user terminal 10 are displayed in each reduced partial area 51.

[0054] In the example of Fig. 9, a partial list area 50 is provided in a part of the screen of the display unit 13 (for example, near the right end). A predetermined number or less (seven or less in the example of Fig. 9) of reduced partial areas 51 are arranged and displayed in the partial list area 50. In the example of Fig. 9, reduced partial areas 51 of all five users participating in a conference room with the theme of casual conversation are displayed.

[0055] If there are more than a predetermined number of participants, the reduced partial areas 51 for the number of participants exceeding the predetermined number are not displayed. However, if the number of reduced partial areas 51 displayed in the part list area 50 is reduced for some reason, the reduced partial areas 51 of users that were not displayed until then may be displayed in the part list area 50. The reduced partial areas 51 displayed in the part list area 50 may be displayed in a scrolled manner. For example, in the screen of FIG. 9, when the display of the part list area 50 is scrolled downward, the reduced partial area 51 at the top end is no longer displayed, and the second to seventh reduced partial areas 51_ that were displayed until then are each moved up by one, and the eighth reduced partial area 51 that was not displayed until then may be displayed at the bottom end.

[0056] FIG. 10 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of FIG. 10, an image of a user displayed in the reduced partial area 51_1 is displayed in the partial area 52_1. When a user performs a predetermined operation (e.g., touch, click, etc.: hereinafter referred to as an "enlarged display operation") on the reduced partial area 51 displayed on the display unit 13 of the user terminal 10, the operated reduced partial area 51 is displayed as a partial area 52. The partial area 52 has a larger area than the reduced partial area 51. In the partial area 52, the same image as that displayed in the reduced partial area 51 is enlarged and displayed. For example, one or more partial areas 52 are displayed in an area other than the partial list area 50. The reduced partial area 51 corresponding to the user displayed in the partial area 52 may be deleted from the partial list area 50. In the example of FIG. 10, an operation is performed on the reduced partial area 51_1 originally displayed in the partial list area 50, so that the partial area 52_1 is displayed. At that time, the reduced partial area 51_1 is deleted from the partial list area 50. As a result of this deletion, the display positions of the reduced partial regions 51_2 to 51_5 are moved upward one by one.

[0057] FIG. 11 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of FIG. 11, the images of the users displayed in the reduced partial areas 51_1 and 51_4 are displayed in the partial areas 52_1 and 52_4, respectively. Such a display is displayed as a result of, for example, performing an operation on the reduced partial areas 51_1 and 51_4 originally displayed in the partial list area 50. When a plurality of partial areas 52 are displayed, the sizes of the partial areas 52 may be displayed as the same size, or some or all of the partial areas 52 may be displayed as being different in size from the sizes of the other partial areas 52. In the example of FIG. 11, the partial areas 52 are displayed as the same size.

[0058] Fig. 12 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of Fig. 12, as a result of an operation being performed on all reduced partial areas 51 displayed in the partial list area 50, partial areas 52 are displayed for all users. When partial areas 52 are displayed for all users, the partial list area 50 does not need to be provided. In this case, the area that was used as the partial list area 50 is also used as a display area for the partial areas 52. This makes it possible to display the partial areas 52 more widely.

[0059] As shown in Figs. 10 to 12, a partial area individual operation button and a partial area whole operation button 505 are displayed on the display unit 13. The partial area individual operation button includes an individual enlargement button 53 and a reduction button 54. The partial area individual operation button is displayed for each partial area 52. One partial area whole operation button 505 is displayed on the screen of the display unit 13. When the individual enlargement button 53 is operated, the partial area 52 is enlarged and displayed as an area larger than the other partial areas 52. At this time, the other partial areas 52 do not have to be displayed. When the partial area 52 is enlarged and displayed, a back button is also displayed. When the back button is operated, the display state returns to the state immediately before the individual enlargement button 53 was operated. When the reduction button 54 is operated, the partial area 52 for that user is erased, and a reduced partial area 51 is displayed in the partial list area 50.

[0060] The partial area whole operation button 505 may include a full screen button, a close all button, and a hide button. These buttons may be displayed by operating the partial area whole operation button 505, for example. When the full screen button is operated, full screen display is performed with the partial area 52 still displayed. When the close all button is operated, all partial areas 52 are erased, and all reduced partial areas 51 are displayed in the partial list area 50 as much as possible. That is, the display returns to a state in which none of the partial areas 52 is displayed. When the hide button is operated, the display of all partial areas 52 is temporarily erased, and the conference room area image 30 originally displayed in the background is displayed. In this display state, operations on the conference room area image 30 and the icon image 41 are valid. On the other hand, when even one partial area 52 is displayed, operations on the conference room area image 30 and the icon image 41 are invalid. When the hide button is operated, the back button is also displayed. When the back button is operated, the display returns to the state immediately before the hide button was operated.

[0061] 13 is a diagram showing a specific example of a state in which screen sharing is being performed in the reduced partial area 51. In the reduced partial area 51_10, the reduced partial area 51_11, and the reduced partial area 51_12, as a result of a screen sharing operation being performed by a user corresponding to the reduced partial area 51, a moving image of the shared screen is displayed as a provided moving image. In this manner, screen sharing may be performed by a plurality of users at the same time.

[0062] Fig. 14 is a diagram showing a specific example of a state in which screen sharing is being performed in the reduced partial areas 51 and the partial areas 52. In the example of Fig. 14, as a result of a predetermined operation being performed in the reduced partial areas 51_10 and the reduced partial areas 51_12, the images of the shared screens displayed in the reduced partial areas 51_10 and 51_12 are displayed in the partial areas 52_10 and 52_12, respectively. In this manner, screen sharing being performed in parallel by different users may be displayed in a plurality of partial areas 52.

[0063] Next, a process of determining the display contents of the user terminal 10 will be described. A specific example of the display contents to be determined is information indicating whether the display area of ​​each user is to be allocated to the reduced partial area 51 or the partial area 52 (hereinafter referred to as "area allocation information"). The area allocation information may include information indicating the user to be allocated to the reduced partial area 51 and information indicating the user to be allocated to the partial area 52. The area allocation information may be determined, for example, by the display control unit 181 of the user terminal 10, or may be determined by the display information generating unit 233 of the conversation control device 20. In the following description, a case where the display control unit 181 of the user terminal 10 determines the area allocation information will be described as an example. The display control unit 181 generates an image to be displayed on the display unit 13 based on the determined area allocation information.

[0064] In determining the area allocation information, there may be a plurality of display control modes. For example, three modes, a manual mode, an automatic mode, and a focus mode, may be defined. In each user terminal 10, the display control mode may be set by the user, or the display control mode may be set for each conference room.

[0065] In the manual mode, the area allocation information is determined according to the operation of the user of the user terminal 10. For example, in the initial state, all users connected to the same conference room (hereinafter referred to as "connected users") are assigned to the reduced partial area 51 as shown in Fig. 9. Thereafter, when an operation (enlarged display operation) is performed in the reduced partial area 51 to display that user as the reduced partial area 52, that user is assigned to the reduced partial area 52. For example, a process is performed to change the display from Fig. 9 to that of Fig. 10. Also, when an operation (for example, an operation of the reduce button 54) is performed in the partial area 52 to display that user as the reduced partial area 51, that user is assigned to the reduced partial area 51.

[0066] In the automatic mode, the users who are transmitting the provided video are assigned to the partial area 52, and the users who are not transmitting the provided video are assigned to the reduced partial area 51. FIG. 15 is a diagram showing a display example in the automatic mode. In FIG. 15, three users are transmitting captured image data (specific examples of the provided video), and two users are not transmitting the provided video. Therefore, the three users who are transmitting the provided video are assigned to the partial area 52, and the two users who are not transmitting the provided video are assigned to the reduced partial area 51. In FIG. 15, partial areas 52_1, 52_4, and 52_5 are displayed as the partial areas 52 of the three users who are transmitting the provided video. In addition, reduced partial areas 51_2 and 51_3 are displayed as the reduced partial areas 51 of the two users who are not transmitting the provided video.

[0067] In the automatic mode, the above-mentioned allocation is performed as an initial state, and thereafter, the allocation may be changed according to the user's operation. For example, when an operation for displaying the user as the reduced partial area 51 in the partial area 52 (for example, operation of the reduce button 54) is performed, the user may be assigned to the reduced partial area 51. Specifically, it is as follows. When the reduce button 54 of the partial area 52_4 is operated in the state of FIG. 15, it is displayed as shown in FIG. 16. That is, the user of the partial area 52_4 for which the reduce button 54 is operated is assigned to the reduced partial area 51.

[0068] For example, when an operation is performed to further enlarge and display the user in the partial area 52 (for example, by operating the individual enlargement button 53), only that user is displayed in the larger partial area 52, and other users are not displayed as partial areas 52. Specifically, it is as follows. When the individual enlargement button 53 of the partial area 52_1 is operated in the state of FIG. 15, it is displayed as shown in FIG. 17. That is, only the user of the partial area 52_1 for which the individual enlargement button 53 was operated is assigned to the partial area 52, and all other users are assigned to the reduced partial area 51. Therefore, after the operation, only the partial area 52_1 is displayed in the large area. And the images of other users are displayed as the reduced partial area 51, not the partial area 52, even if they are users who are transmitting the provided video.

[0069] 17, in the partial area 52 displayed by operating the individual enlargement button 53, an individual enlargement stop button 55 may be displayed instead of the individual enlargement button 53. When the individual enlargement stop button 55 is operated, the users who are currently transmitting provided videos are assigned to the partial area 52, and the users who are not currently transmitting provided videos are assigned to the reduced partial area 51, and a new screen is generated.

[0070] In the automatic mode, the allocation may be changed by dividing the users into target users and non-target users. The target users are users who are the targets of the determination of whether or not to allocate them to the partial area 52. The non-target users are users who are not the targets of the determination of whether or not to allocate them to the partial area 52, and are users who are not displayed in the partial area 52. In the automatic mode, all connected users are set as target users in the initial state. Therefore, in the initial state, all users who are transmitting provided videos are displayed in the partial area 52 as shown in FIG. 15, and only users who are not transmitting provided videos are displayed in the reduced partial area 51. When the reduction button 54 is operated in the partial area 52, the users in the partial area 52 are set as non-target users rather than target users. In that case, even if the users set as non-target users transmit provided videos, they are not displayed in the partial area 52 but are displayed in the reduced partial area 51. In addition, when transitioning from the manual mode to the automatic mode, the users displayed in the partial area 52 in the manual mode immediately before the transition may be set as target users in the initial state of the automatic mode immediately after the transition.

[0071] Specifically, it is as follows. When the reduce button 54 of the partial area 52_1 is operated in the state of FIG. 15, it is displayed as shown in FIG. 18. That is, the user of the partial area 52_1 for which the reduce button 54 is operated is set as a non-target user. Therefore, even if the user of the partial area 52_1 is sending a provided video, the user is assigned to the reduced partial area 51. Therefore, after the operation, the user of the partial area 52_1 is displayed in the reduced partial area 51_1. On the other hand, among the users set as target users, the users who are sending a provided video are displayed in the partial areas 52 (52_4 and 52_5).

[0072] A user set as a non-target user may be set as a target user again by subsequently satisfying a predetermined condition. For example, the user may be set as a target user again on the condition that the user once stops transmitting the provided video and then starts transmitting the provided video again. For example, the user may be set as a target user again on the condition that an enlargement display operation is performed in the reduced partial area 51 of the user. For example, all connected users including non-target users may be set as target users on the condition that an operation for simultaneously setting all connected users as target users (for example, an operation of an all enlargement button defined in this way) is performed.

[0073] An all enlargement stop button may be displayed. When the all enlargement stop button is operated, all users are set to non-target users. In this case, all users are displayed in the reduced portion area 51 regardless of whether they are transmitting a provided video image or not. This concludes the explanation of the automatic mode.

[0074] Next, the focus mode will be described. In the focus mode, only one user at most is displayed in the partial area 52. Other users are displayed, for example, in the reduced partial area 51. FIG. 19 is a diagram showing an example of a display in an initial state set in the focus mode. At this point, since no user is selected as a focus target, the partial area 52 is not displayed, and all connected users are displayed in the reduced partial area 51. When an enlargement display operation is performed in the reduced partial area 51 in the state of FIG. 19, only that user is displayed in the partial area 52. For example, when an enlargement display operation is performed on a user in the reduced partial area 51_3 in FIG. 19, only this user is assigned to the partial area 52 (52_3). FIG. 20 is a diagram showing an example of a display after such an operation.

[0075] In the focus mode, when an enlargement operation is performed in the reduced partial area 51 while the partial area 52 is displayed, only the user for whom the enlargement operation is performed is assigned to the partial area 52, and the user who was previously displayed in the partial area 52 is assigned to the reduced partial area 51. For example, in FIG. 20, when an enlargement operation is performed in the reduced partial area 51_2, the display transitions as shown in FIG. 21. That is, the user who was displayed in the partial area 52_3 is displayed in the reduced partial area 51_3, and the user of the reduced partial area 51_2 for which the enlargement operation is performed is displayed in the partial area 52_2. In addition, when transitioning from the focus mode to the automatic mode, the user displayed in the partial area 52 in the focus mode immediately before the transition may be set as the target user in the initial state of the automatic mode immediately after the transition.

[0076] The above-mentioned processes are executed by, for example, the display control unit 181. The same processes are executed by the display information generation unit 233. The display control unit 181 that executes the above-mentioned processes may be executed by, for example, an application that has been installed in advance in the user terminal 10. The display control unit 181 that executes the following processes may be realized by downloading an application for executing the following processes from a server such as the conversation control device 20 and executing the application every time an information processing device operated by a user operates as the user terminal 10 of the conversation control system 100.

[0077] The conversation control system 100 thus configured can realize a screen display that is more convenient for the user when talking with one or more other users. Specifically, it is as follows. In the automatic mode of the conversation control system 100, whether to display an image (e.g., text, still image, or video) related to each user in the reduced partial area 51 or the partial area 52 is determined based on whether or not a provided video has been sent from that user. For example, a user who sends a provided video is displayed in a relatively larger area (e.g., the partial area 52) compared to a user who does not send a provided video. The provided video is more likely to contain more useful information than simply text or still images. Therefore, by preferentially displaying the provided video in a larger area, it is possible to realize a screen display that is more convenient for the user.

[0078] Furthermore, the conversation control system 100 can set users who are to be displayed in a larger area (target users) and users who are not to be displayed (non-target users) among the connected users. This makes it possible to realize a screen display that is more convenient for users.

[0079] In addition, data of the provided video may be transmitted from the user terminal 10 that provides the video (e.g., the sharing source) to the user terminal 10 that receives the video via the conversation control device 20, or it may be transmitted from the user terminal 10 that provides the video to the user terminal 10 that receives the video without passing through the conversation control device 20.

[0080] The conversation control unit 183 or the conversation control unit 234 may perform control so that the voice of the user displayed in the partial area 52 is output louder than the voice of the user displayed in the reduced partial area 51.

[0081] FIG. 22 is a diagram showing an outline of a hardware configuration example of an information processing device 900 applied to this embodiment. The information processing device 900 includes a processor 901, a main memory device 902, a communication interface 903, an auxiliary memory device 904, an input / output interface 905, and an internal bus 906. The processor 901, the main memory device 902, the communication interface 903, the auxiliary memory device 904, and the input / output interface 905 are communicably connected to each other via the internal bus 906. The information processing device 900 may be applied to, for example, a user terminal 10 and a conversation control device 20. In this case, for example, the communication unit 11 and the communication unit 21 may be configured using the communication interface 903. For example, the memory unit 17 and the memory unit 22 may be configured using the auxiliary memory device 904. In addition, the control unit 18 and the control unit 23 may be configured using the processor 901 and the main memory device 902.

[0082] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0083] 100...conversation control system, 10...user terminal, 20...conversation control device, 11...communication unit, 12...operation unit, 13...display unit, 14...audio input unit, 15...audio output unit, 16...imaging unit, 17...storage unit, 171...user information storage unit, 18...control unit, 181...display control unit, 182...conference control unit, 183...conversation control unit, 21...communication unit, 22...storage unit, 221...conference room information storage unit, 23...control unit, 231...user control unit, 232...conference room control unit, 233...display information generation unit, 234...conversation control unit, 30...conference room area image, 31...conversation area, 311...conference room name, 312...entry button, 313...detail button, 32...listening area, 41...icon image, 91...outdoor area

Claims

1. a control unit that displays, in a virtual conference room for a user and other users to converse with each other via a network, a partial area and a reduced partial area capable of displaying an image provided from a user terminal of one or more other users participating in the same conference room as the user on a screen of the user's user terminal, The reduced partial area is an area that is relatively smaller than the partial area, The control unit assigns users who are transmitting moving images to the partial area, and assigns users who are not transmitting moving images to the reduced partial area for display.

2. The control unit sets target users to be assigned to the partial area for users who are participating in the same conference room as the user, and assigns and displays users who are not set as target users to the reduced partial area even if they are sending moving images.

3. The control system according to claim 2 , wherein the control unit removes the user from the target users when an operation indicating removal from the target users is performed by the user.

4. a control step of the computer displaying, in a virtual conference room for a user and other users to talk to each other via a network, a partial area and a reduced partial area capable of displaying an image provided from a user terminal of one or more other users participating in the same conference room as the user on a screen of the user terminal of the user, The reduced partial area is an area that is relatively smaller than the partial area, In the control step, a user who is transmitting a moving image is assigned to the partial area, and a user who is not transmitting a moving image is assigned to the reduced partial area for display.

5. a control unit that displays, in a virtual conference room for a user and other users to converse with each other via a network, a partial area and a reduced partial area capable of displaying an image provided from a user terminal of one or more other users participating in the same conference room as the user on a screen of the user's user terminal, The reduced partial area is an area that is relatively smaller than the partial area, The control unit is a computer program for causing a computer to function as a terminal device that assigns users who are sending moving images to the partial area and assigns users who are not sending moving images to the reduced partial area for display.

Images