Control system, control method, and computer program

Abstract

One aspect of the present invention is a control system comprising a communication unit that receives data pertaining to a plurality of moving images provided from another device, and a control unit that allocates and displays the moving images received by the communication unit to a plurality of partial regions obtained by dividing a display screen into a plurality of sections. The control unit determines the allocation of the partial regions and the moving images so that the sum of the areas of the display regions of the plurality of moving images increases.

Description

Control system, control method, and computer program

[0001] The present invention relates to a technique for receiving and displaying multiple moving images online. This application claims priority to Japanese Patent Application No. 2023-096540, filed on June 12, 2023, the contents of which are incorporated herein by reference.

[0002] Systems for conducting conversations and conferences over a network have been proposed. In these systems, facial images of participants captured by a camera and images of windows shared by participants are generally displayed in partial areas set up for each user. For example, Patent Literature 1 describes a technique for extracting an image of a captured user's facial area and displaying it with a blurred background.

[0003] Japanese Patent Application Laid-Open No. 2012-213013

[0004] Conventionally, when multiple members are participating, there have been systems in which a partial area is provided for each of one or more participating members, and faces and moving images are displayed therein. Some systems allow users to select from multiple predefined screen layouts for the screen layout of each partial area for each member. For example, there is a screen layout in which the partial areas of one or more users are set to the same size, and each user's image is displayed within the partial area. Another example is a screen layout in which the partial area of ​​a representative user (e.g., the user currently speaking) is set larger than the partial areas of other users. Another example is a screen layout in which a partial area for displaying a shared window or desktop image is set larger than the user's partial area. However, sufficient convenience has not always been achieved. This problem is not necessarily limited to systems for conversations or conferences, but is a common problem in systems that display multiple moving images provided by other users or other systems.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique that enables a screen display that is more convenient for users to view in a system that provides a plurality of moving images.

[0006] One aspect of the present invention is a control system comprising a communication unit that receives data on multiple moving images provided from another device, and a control unit that allocates and displays the moving images received by the communication unit to multiple partial areas obtained by dividing a display screen into multiple parts, wherein the control unit determines the allocation of the partial areas to the moving images so as to maximize the sum of the areas of the display areas for the multiple moving images.

[0007] In one aspect of the present invention, in the control system described above, the control unit determines the allocation of the partial regions to the moving images based on the aspect ratio of each moving image and the aspect ratio of each partial region.

[0008] One aspect of the present invention is the above-mentioned control system, wherein the control unit determines the allocation of the partial regions to the moving images so that the order of the aspect ratios of each moving image matches the order of the aspect ratios of each partial region.

[0009] One aspect of the present invention is a control method in which a computer receives data of multiple moving images provided from another device, the computer allocates and displays the received moving images to multiple partial areas obtained by dividing a display screen into multiple parts, and the computer determines the allocation of the partial areas to the moving images so as to maximize the sum of the areas of the display areas of the multiple moving images.

[0010] One aspect of the present invention is a computer program for operating a computer as a control system that includes a communication unit that receives data on multiple moving images provided from another device, and a control unit that allocates and displays the moving images received by the communication unit to multiple partial areas obtained by dividing a display screen into multiple parts, wherein the control unit determines the allocation of the partial areas to the moving images so that the sum of the areas of the display areas for the multiple moving images is largest.

[0011] According to the present invention, in a system in which a plurality of moving images are provided, it is possible to realize a screen display that is more convenient for the user.

[0012] 1 is a schematic block diagram showing the system configuration of a conversation control system 100 of the present invention. FIG. 1 is a schematic block diagram showing a specific example of the functional configuration of a user terminal 10. FIG. 2 is a schematic block diagram showing a specific example of the functional configuration of a conversation control device 20. FIG. 3 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. FIG. 4 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. FIG. 5 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. FIG. 6 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. FIG. 7 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. FIG. 8 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. FIG. 9 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. FIG. 10 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. FIG. 11 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. FIG. 1 is a diagram showing an example of the arrangement of a plurality of partial areas 52 displayed on the display unit 13 of the user terminal 10. FIG. 2 is a diagram showing an example of the arrangement of a plurality of partial areas 52 displayed on the display unit 13 of the user terminal 10. FIG. 3 is a diagram showing an example of the arrangement of a plurality of partial areas 52 displayed on the display unit 13 of the user terminal 10. FIG. 4 is a diagram showing a specific example of a pattern of the shape of a provided moving image. FIG. 5 is a diagram showing a specific example of arrangement candidate information. FIG. 6 is a diagram showing a specific example of arrangement candidate information. FIG. 7 is a diagram showing a specific example of arrangement candidate information of a modified example. FIG. 8 is a diagram showing an outline of an example hardware configuration of an information processing device 900 applied to this embodiment.

[0013] A specific configuration example of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic block diagram showing the system configuration of a conversation control system 100 of the present invention. The conversation control system 100 is a specific example of a system in which multiple moving images are provided. The conversation control system 100 is a system in which users operating user terminals 10 can converse with each other via a network 90. ​​The conversation control system 100 includes multiple user terminals 10 and a conversation control device 20. The multiple user terminals 10 and the conversation control device 20 are communicatively connected via the network 90. ​​The network 90 may be a network using wireless communication or a network using wired communication. The network 90 may also be configured by combining multiple networks.

[0014] 2 is a schematic block diagram showing a specific example of the functional configuration of the user terminal 10. The user terminal 10 is configured using information devices such as a smartphone, tablet, personal computer, portable game console, stationary game console, dedicated device, etc. The user terminal 10 includes a communication unit 11, an operation unit 12, a display unit 13, an audio input unit 14, an audio output unit 15, an imaging unit 16, a memory unit 17, and a control unit 18.

[0015] The communication unit 11 is a communication device. The communication unit 11 may be configured as, for example, a network interface. The communication unit 11 communicates data with other devices via the network 90 in accordance with the control of the control unit 18. The communication unit 11 may be a device that performs wireless communication or a device that performs wired communication.

[0016] The operation unit 12 is configured using existing input devices such as a keyboard, a pointing device (mouse, tablet, etc.), buttons, and a touch panel. The operation unit 12 is operated by a user when inputting user instructions to the user terminal 10. The operation unit 12 may be an interface for connecting an input device to the user terminal 10. In this case, the operation unit 12 inputs an input signal generated in the input device in response to a user input to the user terminal 10. The operation unit 12 may be configured using a microphone and a voice recognition device. In this case, the operation unit 12 performs voice recognition on words spoken by the user and inputs character string information of the recognition result to the user terminal 10. In this case, the voice recognition process may be performed by the control unit 18. In this case, the microphone constituting the operation unit 12 may be configured integrally with the voice input unit 14 described below. The operation unit 12 may be configured in any manner as long as it is capable of inputting user instructions to the user terminal 10.

[0017] The display unit 13 is an image display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 13 displays image data used in the conversation control system 100. The display unit 13 may be an interface for connecting an image display device to the user terminal 10. In this case, the display unit 13 generates a video signal for displaying the image data and outputs the video signal to the image display device connected to the display unit 13.

[0018] The voice input unit 14 is configured using a microphone. The voice input unit 14 may be configured as a microphone itself, or may be configured as an interface for connecting a microphone as an external device to the user terminal 10. The microphone acquires the speech of the user who is having a conversation. The voice input unit 14 outputs data of the voice acquired by the microphone to the control unit 18.

[0019] The audio output unit 15 is configured using an audio output device such as a speaker, headphones, or earphones. The audio output unit 15 may be configured as the audio output device itself, or may be configured as an interface for connecting the audio output device to the user terminal 10 as an external device. It is desirable that the audio output device outputs audio so that the user having a conversation can hear the audio. The audio output unit 15 outputs audio corresponding to the audio signal output by the control unit 18.

[0020] The imaging unit 16 is configured using an imaging device such as a camera. The imaging unit 16 may also be an interface for connecting an imaging device such as a camera to the user terminal 10. The imaging unit 16 is preferably installed in a position where it can capture an image of the user's face. In this case, the imaging unit 16 captures an image of the user's face and generates still image or video data (hereinafter referred to as "captured image data"). The imaging unit 16 outputs the generated captured image data to the control unit 18. The captured image data is one form of provided video. The provided video is a video provided from a user terminal 10 or an information device to another user terminal 10. Specific examples of provided video include a video (captured image data) captured by the imaging unit 16 of the user terminal 10 and a video showing a screen shared on the user terminal 10 (hereinafter referred to as a "shared screen"). The shared screen is, for example, a window selected by the user, a window displaying a selected file, a selected desktop, or the like. The video showing the shared screen may reflect operations performed by the user on the screen at any time. For example, when a user operates the position of a cursor, the display position of the cursor changes on the shared screen in response to the operation.

[0021] The storage unit 17 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 17 stores data used by the control unit 18. The storage unit 17 may function as, for example, a user information storage unit 171. The user information storage unit 171 stores information about the user operating the user terminal 10 (hereinafter referred to as "user information"). The user information may include, for example, the user's handle name, user icon data, and user attribute information. The handle name is the name (nickname or real name) used by the user when having a conversation in the conversation control system 100. The icon data is image data used by the user when having a conversation in the conversation control system 100. The attribute information may include, for example, information about the user's age, gender, and affiliation.

[0022] The control unit 18 is configured using a processor such as a CPU (Central Processing Unit) and memory. The control unit 18 functions as a display control unit 181, a conference control unit 182, and a conversation control unit 183 by the processor executing a program. Note that all or part of the functions of the control unit 18 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), a CD-ROM (Compact Disk Read Only Memory), and a semiconductor storage device (e.g., an SSD: Solid State Drive), as well as storage devices such as a hard disk or semiconductor storage device built into a computer system. The above program may be transmitted via a telecommunications line.

[0023] The display control unit 181 receives display information from the conversation control device 20 via the communication unit 11. The display control unit 181 generates an image signal based on the acquired display information and displays it on the display unit 13. At this time, the display control unit 181 controls the display within the display area of ​​the application in which the application of the conversation control system 100 is running. For example, if the application of the conversation control system 100 is a web browser, display control is performed within the web browser window. If the web browser is displayed full-screen, display control is performed on the entire display screen of the display unit 13.

[0024] The display information may be, for example, image data representing the image itself to be displayed. In this case, the entity that generates the image data (image data generation unit) is the display information generation unit 233 of the conversation control device 20. The display information may be, for example, data representing information necessary to generate the image to be displayed (e.g., the number of divided areas to be displayed, the shape of the image area to be displayed in each divided area, the position, size, and display mode of the conference room, and information representing the user's position and icon image). In this case, the display control unit 181 generates image data to be displayed on the display unit 13 based on the display information. In this case, the entity that generates the image data (image data generation unit) is the display control unit 181.

[0025] The conference control unit 182 controls a conference virtually established in the conversation control device 20. For example, when a user operates the operation unit 12 to instruct the establishment of a new conference room, the conference control unit 182 performs processing to establish the new conference room. When the user of the user's own device (user terminal 10) has set up a setting to transmit captured image data to other user terminals 10, the conference control unit 182 transmits captured image data captured by the imaging unit 16 to the other device. The other device serving as the destination may be, for example, the conversation control device 20. In this case, the captured image data is transmitted via the conversation control device 20 to the other user terminals 10 connected to the same conference room as the user's own device (user terminal 10). The other device serving as the destination may be the other user terminals 10 connected to the same conference room as the user's own device (user terminal 10).

[0026] The conversation control unit 183 controls conversations between multiple users participating in the conversation control system 100. A conversation between multiple users may be, for example, a conversation between multiple users participating in a conference room (hereinafter referred to as a "conference room conversation"). When the user of the user's own device (user terminal 10) is participating in a conversation, the conversation control unit 183 controls the conversation between multiple users including the user of the user's own device. In this case, the conversation control unit 183 transmits, for example, audio data input from the audio input unit 14 to the conversation control device 20 via the communication unit 11. When the conversation control unit 183 receives audio data from the conversation control device 20, it outputs the received audio data from the audio output unit 15. When the user of the user's own device (user terminal 10) is not participating in the conversation, the conversation control unit 183 controls the conversation between multiple users excluding the user of the user's own device. In this case, the conversation control unit 183 receives audio data of a conversation between other users and outputs it from the audio output unit 15. The voice data output from the voice output unit 15 may be received from another user terminal 10 via the conversation control device 20, or may be received directly from another user terminal 10 connected in accordance with the control of the conversation control device 20.

[0027] 3 is a schematic block diagram showing a specific example of the functional configuration of the conversation control device 20. The conversation control device 20 is configured using an information processing device such as a personal computer or a server device. The conversation control device 20 includes a communication unit 21, a storage unit 22, and a control unit 23.

[0028] The communication unit 21 is a communication device. The communication unit 21 may be configured as, for example, a network interface. The communication unit 21 communicates data with other devices via the network 90 in accordance with the control of the control unit 23. The communication unit 21 may be a device that performs wireless communication or a device that performs wired communication.

[0029] The storage unit 22 is configured using a storage device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 22 stores data used by the control unit 23. The storage unit 22 may function as a conference room information storage unit 221, for example.

[0030] The conference room information storage unit 221 stores information about conference rooms (hereinafter referred to as "conference room information"). A conference room is a virtual room set up in the conversation control system 100 for users to have conversations. The conference room information may include, for example, information indicating the name set for the conference room. The conference room information may further include information indicating the users currently in the conference room.

[0031] The control unit 23 is configured using a processor such as a CPU and a memory. The control unit 23 functions as a user control unit 231, a conference room control unit 232, a display information generation unit 233, and a conversation control unit 234 by the processor executing a program. Note that all or part of the functions of the control unit 23 may be realized using hardware such as an ASIC, a PLD, or an FPGA. The above program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a semiconductor storage device (e.g., an SSD), and storage devices such as a hard disk or semiconductor storage device built into a computer system. The above program may be transmitted via a telecommunications line.

[0032] The user control unit 231 performs control processing related to users. For example, the user control unit 231 may perform processing for logging in (e.g., authentication processing) for a user who accesses the conversation control device 20.

[0033] The conference room control unit 232 performs control processing related to conference rooms. For example, when the conference room control unit 232 receives an instruction from the user terminal 10 to set up a new conference room, it may generate conference room information based on the received information and register it in the conference room information storage unit 221. When a user performs a predetermined operation to join a conference room, the conference room control unit 232 performs processing to allow the user to join the conference room if predetermined conditions are met. For example, the conference room control unit 232 updates the conference room information storage unit 221 to register that a new user has joined the conference room.

[0034] The display information generation unit 233 generates information (display information) required to generate images to be displayed on the user terminal 10. The display information may include, for example, the number of divided areas, the shape of the image area to be displayed in each divided area, information about the currently installed conference room, and information about the users of each user terminal 10 accessing the conversation control device 20. The display information generation unit 233 transmits the generated display information to the user terminal 10.

[0035] The conversation control unit 234 receives voice data from the user terminals 10. The conversation control unit 234 generates voice data to be output to each user terminal 10 (hereinafter referred to as "conversation voice data") and transmits the conversation voice data to each user terminal 10. The conversation control unit 234 may generate the conversation voice data based on, for example, what kind of conversation the user of each user terminal 10 is participating in. For example, the conversation control unit 234 may transmit, to each user's user terminal 10, conversation voice data including voices received from the user terminals 10 of all users participating in the conference room that the user is in.

[0036] Next, the relationship between the images displayed on the user terminal 10 and the conversational voice data output when each image is displayed will be described.

[0037] 4 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In FIG. 4, the display unit 13 displays a conference room area image 30, a conversation area 31, a listening area 32, an icon image 41, and an operation area image 501. The display unit 13 displays an image of a virtual space in which a conference room can be set up. The virtual space is mainly composed of a conference room area and an area outside the conference room. The conference room area is displayed as the conference room area image 30. The area outside the conference room (hereinafter referred to as the "outdoor area") is the area represented by the symbol 91, and is an area in which icon images 41 of users who are not in the conference room are displayed.

[0038] The conference room area image 30 includes a conversation area 31 and a listening area 32. A user whose icon image 41 is displayed in the conversation area 31 can converse with other users whose icon image 41 is displayed in the conversation area 31 of the same conference room. That is, the voices of the users whose icon images 41 are displayed in the conversation area 31 are transmitted and received between them. In the following description, entering a conference room means that the icon image 41 is displayed in the conversation area 31. That is, entering a conference room means that a user is in a state where they can converse with other users.

[0039] A user whose icon image 41 is displayed in the listening area 32 (hereinafter referred to as a "listening user") can listen to the conversation of the user whose icon image 41 is displayed in the conversation area 31 of the conference room (a user who has entered the conference room). However, the listening user cannot speak in the conference room. In other words, the voice uttered by the listening user is not output to the user terminals 10 of other users who have entered the conference room or to the user terminals 10 of other listening users in the conference room.

[0040] The conference room area image 30 is represented as a geometric image showing an area of ​​a predetermined size. In the example of this embodiment, the conference room area image 30 is represented as a circular image. The name of the conference room 311 is displayed near the center of the conference room area image 30. In the example of FIG. 4, "Chat" is the name of the conference room. By displaying the name of the conference room, even a user located in the outdoor area 91 can roughly guess what kind of conversation is taking place in the conference room without entering the conference room.

[0041] An enter button 312 and a details button 313 may be displayed inside the conference room area image 30. The enter button 312 is a button for requesting entry into the conference room. In response to a user located in the outdoor area 91 operating the enter button 312, the conference room control unit 232 determines that the user is requesting entry into the conference room. In response to the request, the conference room control unit 232 determines whether or not to permit entry, and controls the user's entry in accordance with the determination result.

[0042] The details button 313 is a button for displaying detailed information about the conference room. In response to operating the details button 313, a details display area is displayed on the screen. The details display area displays detailed information about the conference room, as well as buttons and input boxes for performing operations on the conference room. For example, the name of the conference room, a button for changing the name of the conference room, an input box for entering the new name of the conference room, etc. may be displayed.

[0043] The icon image 41 is an image that indicates the virtual position of the user and information about the user. The icon image 41 may be displayed according to icon data included in the user information, for example.

[0044] 5 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of FIG. 5, icon images 41 (41_1 and 41_2) of two users and a conference room area image 30 of one conference room are displayed. No users have yet participated in the conference room represented by the conference room area image 30. The two users have not participated in any conference rooms. Therefore, the icon images 41_1 and 41_2 are displayed in the outdoor area 91, not inside the conference room area image 30.

[0045] A user who wishes to participate in a conference room can do so (enter) the conference room by performing a predetermined operation. For example, a user can join the conference room by operating the entry button 312 in the conference room area image 30 in which the user wishes to participate. Note that the operation for entering the room does not need to be limited to the above. If entry restrictions are set for a conference room, only users who are permitted to enter according to the entry restrictions can enter the conference room.

[0046] 6 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of FIG. 6, one user is participating in a conference room with the theme of chatting. Therefore, an icon image 41 (41_2) of one user is displayed inside the conference room area image 30 indicating the conference room with the theme of chatting. At this point, there are no other users participating in the same conference room as the user of icon image 41_2. Therefore, the user of icon image 41_2 cannot converse with other users in the conference room with the theme of chatting.

[0047] When a user represented by icon image 41_1 in the outdoor area 91 performs an operation to join a conference room with a chat theme, the users participating in the conference room are notified that another user wishes to join. For example, a predetermined sound (e.g., a door knock) may be output from each user's user terminal 10. The users to be notified may be all users participating in the conference room, or only a predetermined subset of users (e.g., the user who is the administrator of the conference room).

[0048] After the notification, if a predetermined condition is met, the user of icon image 41_1 can participate in the chat-themed conference room. The predetermined condition may be, for example, simply the passage of a predetermined time (e.g., 1 second, 3 seconds, 5 seconds, etc.), permission to participate by a predetermined user (e.g., the conference room administrator, a participant, etc.), or the fulfillment of a participation condition that is preset for the conference room. Furthermore, a predetermined condition may not be set. In this case, all users who wish to participate can participate in the conference room (e.g., a chat-themed conference room).

[0049] FIG. 7 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of FIG. 7, two users are participating in a conference room with the theme of chatting. Therefore, icon images 41 (41_1 and 41_2) of the two users are displayed inside the conference room area image 30 indicating the conference room with the theme of chatting. At this point, the user of icon image 41_1 and the user of icon image 41_2 are participating in the same conference room. Therefore, the user of icon image 41_1 and the user of icon image 41_2 can have a conversation in the conference room with the theme of chatting. In other words, users participating in the same conference room can have a conversation using the user terminal 10.

[0050] More specifically, the process is as follows: The user terminal 10 of a user participating in a conference room receives and outputs voice data spoken by other users participating in the same conference room. The user terminal 10 of a user participating in a conference room also transmits the voice data spoken by that user to the conversation control device 20. This transmission causes the voice data to be output at the user terminals 10 of the other users participating in the same conference room. When a user represented by icon image 41_3 in the outdoor area 91 performs an operation to participate in a conference room with a chat theme, after certain conditions are met, the user represented by icon image 41_3 can participate in the conference room with a chat theme.

[0051] FIG. 8 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of FIG. 8, three users are participating in a conference room with the theme of chatting. Therefore, icon images 41 (41_1, 41_2, and 41_3) of the three users are displayed inside the conference room area image 30 indicating the conference room with the theme of chatting. At this point, the user of icon image 41_1, the user of icon image 41_2, and the user of icon image 41_3 are participating in the same conference room. Therefore, the user of icon image 41_1, the user of icon image 41_2, and the user of icon image 41_3 can have a conversation in the conference room with the theme of chatting. The same applies when four or more users participate in the same conference room.

[0052] FIG. 9 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of FIG. 9, six users are participating in a conference room with a theme of casual conversation. In the example of FIG. 9, the user terminal 10 is operating in video conversation mode. In video conversation mode, a reduced partial area 51 is displayed for each user participating in the same conference room (the conference room with a theme of casual conversation in FIG. 9). The reduced partial area 51 is an area in which a video image (provided video image) provided from the user terminal 10 of the user associated with that reduced partial area 51 can be displayed.

[0053] Other types of moving images may be displayed as provided moving images in the reduced partial area 51. In the example of Fig. 9, each reduced partial area 51 displays a provided moving image including the face of each user captured by the imaging unit 16 of the user terminal 10.

[0054] 9, a partial list area 50 is provided in a portion of the screen of the display unit 13 (for example, near the right edge). A predetermined number or less (seven or less in the example of FIG. 9) of reduced partial areas 51 are arranged and displayed in the partial list area 50. In the example of FIG. 9, reduced partial areas 51 for all five users participating in a conference room with a chat theme are displayed.

[0055] If the number of participants exceeds a predetermined number, reduced portion areas 51 for the number of participants exceeding the predetermined number will not be displayed. However, if the number of reduced portion areas 51 displayed in the portion list area 50 is reduced for some reason, reduced portion areas 51 of users that were not previously displayed may be displayed in the portion list area 50. Furthermore, the reduced portion areas 51 displayed in the portion list area 50 may be scrolled. For example, on the screen of FIG. 9 , if the display of the portion list area 50 is scrolled downward, the reduced portion area 51 at the top may disappear, the second to seventh reduced portion areas 51_ that were previously displayed may each move up one position, and the eighth reduced portion area 51 that was not previously displayed may be displayed at the bottom.

[0056] FIG. 10 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of FIG. 10 , an image of a user that was displayed in reduced partial area 51_1 is displayed in partial area 52_1. When a user performs a predetermined operation (e.g., touch, click, etc.) on reduced partial area 51 displayed on the display unit 13 of the user terminal 10, the operated reduced partial area 51 is displayed as partial area 52. Partial area 52 has a larger area than reduced partial area 51. In partial area 52, the same image that was displayed in reduced partial area 51 is displayed enlarged. One or more partial areas 52 may be displayed in an area other than the partial list area 50, for example. The reduced partial area 51 corresponding to the user displayed in partial area 52 may be deleted from the partial list area 50. In the example of FIG. 10 , an operation was performed on reduced partial area 51_1 that was originally displayed in the partial list area 50, so partial area 52_1 is displayed. At that time, reduced partial area 51_1 was deleted from the partial list area 50. As a result of this deletion, the display positions of the reduced partial areas 51_2 to 51_5 are moved upward one by one.

[0057] FIG. 11 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of FIG. 11, the user images displayed in reduced partial areas 51_1 and 51_4 are displayed in partial areas 52_1 and 52_4, respectively. Such displays are displayed, for example, as a result of operations being performed on reduced partial areas 51_1 and 51_4 that were originally displayed in the partial list area 50. When multiple partial areas 52 are displayed, each partial area 52 may be displayed with the same size, or some or all of the partial areas 52 may be displayed with sizes different from the other partial areas 52. In the example of FIG. 11, each partial area 52 is displayed with the same size.

[0058] 12 is a diagram showing a specific example of an image displayed on the display unit 13 of the user terminal 10. In the example of FIG. 12, as a result of operations being performed on all reduced partial areas 51 displayed in the partial list area 50, partial areas 52 are displayed for all users. If partial areas 52 are displayed for all users, the partial list area 50 does not need to be provided. In this case, the area that was previously used as the partial list area 50 is also used as the display area for the partial areas 52. This makes it possible to display the partial areas 52 more widely.

[0059] As shown in Figures 10 to 12, the display unit 13 displays a partial area individual operation button and an entire partial area operation button 505. The partial area individual operation button includes an individual enlargement button 53 and a reduction button 54. The partial area individual operation button is displayed for each partial area 52. One entire partial area operation button 505 is displayed on the screen of the display unit 13. When the individual enlargement button 53 is operated, the partial area 52 is enlarged and displayed as an area larger than the other partial areas 52. At this time, the other partial areas 52 do not need to be displayed. When a partial area 52 is enlarged and displayed, a back button is also displayed. When the back button is operated, the display returns to the state it was in immediately before the individual enlargement button 53 was operated. When the reduction button 54 is operated, the partial area 52 for that user is erased, and a reduced partial area 51 is displayed in the partial list area 50.

[0060] The partial area entire operation buttons 505 may include a full screen button, a close all button, and a hide button. These buttons may be displayed by operating the partial area entire operation button 505, for example. When the full screen button is operated, a full screen display is performed while the partial areas 52 remain displayed. When the close all button is operated, all partial areas 52 are erased, and as many reduced partial areas 51 as possible are displayed in the partial list area 50. In other words, the display returns to a state where no partial areas 52 are displayed. When the hide button is operated, the display of all partial areas 52 is temporarily erased, and the conference room area image 30 originally displayed in the background is displayed. In this display state, operations on the conference room area image 30 and the icon image 41 are valid. On the other hand, if even one partial area 52 is displayed, operations on the conference room area image 30 and the icon image 41 are invalid. When the hide button is operated, a back button is also displayed. When the back button is operated, the display returns to the state immediately before the hide button was operated.

[0061] 13 is a diagram showing a specific example of a state in which screen sharing is being performed in reduced-size portion area 51. In reduced-size portion area 51_10, reduced-size portion area 51_11, and reduced-size portion area 51_12, a moving image of the shared screen is displayed as a provided moving image as a result of a screen sharing operation performed by a user corresponding to reduced-size portion area 51. In this way, screen sharing may be performed by multiple users at the same time.

[0062] 14 is a diagram showing a specific example of a state in which screen sharing is being performed in reduced partial areas 51 and 52. In the example of Fig. 14, as a result of a predetermined operation being performed in reduced partial areas 51_10 and 51_12, the images of the shared screens displayed in reduced partial areas 51_10 and 51_12 are displayed in partial areas 52_10 and 52_12, respectively. In this way, screen sharing being performed in parallel by different users may be displayed in multiple partial areas 52.

[0063] Next, we will explain the process of determining the display content of the user terminal 10. Specific examples of the display content to be determined include information indicating the size and display position of the partial areas 52, the allocation of provided moving images to each partial area 52, and the size and display position of the image display area 56 in each partial area 52. This determination process may be performed, for example, by the display control unit 181 of each user terminal 10, or by the display information generation unit 233 of the conversation control device 20.

[0064] In the following explanation, an example will be described in which the display content determination process is performed by the display control unit 181 of each user terminal 10, but the process is similar when performed by the display information generation unit 233. Note that the display control unit 181 that executes the following process may be executed by, for example, an application that is pre-installed in the user terminal 10. Furthermore, the display control unit 181 that executes the following process may be realized by downloading an application for executing the following process from a server such as the conversation control device 20 and executing the application each time an information processing device operated by a user operates as the user terminal 10 of the conversation control system 100.

[0065] FIG. 15 is a diagram showing an example of the configuration of a partial area 52. An image display candidate area 55 is provided inside the partial area 52. The image display candidate area 55 indicates the largest area of ​​the partial area 52 in which the provided moving image can be displayed. In each partial area 52, a provided moving image of the same shape (size) as the image display candidate area 55 can be displayed as the largest image size. The image display candidate area 55 may be the same shape and size as the partial area 52, or may be an area smaller than the partial area 52. In the example of FIG. 15, the image display candidate area 55 has a height H1 and a width W1.

[0066] An image display area 56 is set inside the image display candidate area 55. The image display area 56 is an area for displaying the provided video transmitted from the user or device assigned to the partial area 52. In FIG. 15 , the image display area 56 has a height H2 and a width W2. The height H2 and width W2 of the image display area 56 are determined based on the height H3 and width W3 of the provided video transmitted from the user or device assigned to the partial area 52. W2 and H2 are determined so that the ratio of W2 to H2 is the same as the ratio of W3 to H3. In other words, the shape of the image display area 56 is determined so that its aspect ratio is the same as the aspect ratio of the provided video. For example, in each partial area 52, the image display area 56 may be determined so that the area of ​​the image display area 56 for displaying the provided video assigned to that partial area 52 is maximized. The display control unit 181 may obtain the aspect ratio of the finalized common image by any method. For example, the display control unit 181 may receive a provided video from another device and acquire the aspect ratio based on the data of the received provided video. For example, when providing a video from its own device to another device, the display control unit 181 may be configured to transmit information indicating the aspect ratio of the video (provided video) to the other device. In this case, the display control unit 181 of each device acquires information indicating the aspect ratio from the device that is the provider of the provided video.

[0067] FIG. 16 is a diagram showing an example of the arrangement of multiple partial regions 52 displayed on the display unit 13 of the user terminal 10. Multiple arrangement options are defined depending on the number of partial regions 52 displayed on the display unit 13. The arrangement options are expressed, for example, as [m1, ..., mn: horizontal] and [m1, ..., mn: vertical], where m and n are integers equal to or greater than 1. In the [m1, ..., mn: horizontal] arrangement, the screen of the display unit 13 is divided horizontally into n rows, and m1 partial regions 52 are arranged in the first row counting from the top, m2 partial regions 52 in the second row, and mn partial regions 52 in the nth row, all of which are the same size. In the [m1, ..., mn: vertical] arrangement, the screen of the display unit 13 is divided vertically into n columns, and m1 partial regions 52 are arranged in the first row counting from the left, m2 partial regions 52 in the second row, and mn partial regions 52 in the nth row, all of which are the same size.

[0068] In FIG. 16 , n = 3, m1 = 1, m2 = 2, and m3 = 2, and the division direction is horizontal. That is, the arrangement candidate in FIG. 16 is [1, 2, 2: horizontal]. Since n = 3, the display unit 13 is divided into three rows. m1 = 1 partial region 52 (52_a) is arranged in the first row, m2 = 2 partial regions 52 (52_b, 52_c) is arranged in the second row, and m3 = 2 partial regions 52 (52_d, 52_e) is arranged in the third row. In this embodiment, when the arrangement candidate is horizontal, when the screen of the display unit 13 is divided into n rows, the rows are equally divided so that the vertical length of each row is the same. Furthermore, in this embodiment, the partial regions 52 with the largest area are arranged from top to bottom. However, the vertical length of each row and the relationship between the area and arrangement of each partial region 52 are merely examples and need not be limited to those described above.

[0069] FIG. 17 is a diagram showing an example of the arrangement of multiple partial regions 52 displayed on the display unit 13 of the user terminal 10. In FIG. 17, n = 3, m1 = 1, m2 = 2, and m3 = 2, and the division direction is vertical. That is, the arrangement candidate in FIG. 17 is [1, 2, 2: vertical]. Since n = 3, the display unit 13 is divided into three columns. m1 = 1 partial region 52 (52_a) is arranged in the first column, m2 = 2 partial regions 52 (52_b, 52_c) is arranged in the second column, and m3 = 2 partial regions 52 (52_d, 52_e) is arranged in the third column. In this embodiment, when the arrangement candidate is vertical, when the screen of the display unit 13 is divided into n columns, the columns are equally divided so that the horizontal length of each column is the same. Furthermore, in this embodiment, the partial regions 52 with larger areas are arranged from left to right. However, the horizontal length of each row and the relationship between the area and arrangement of each partial region 52 are merely examples, and are not limited to those described above.

[0070] In Fig. 18, n = 2, m1 = 2, m2 = 3, and the partition direction is horizontal. That is, the layout candidate in Fig. 18 is [2, 3: horizontal]. Because n = 2, the display unit 13 is divided into two rows. m1 = 2 partial regions 52 (52_a, 52_b) are arranged in the first row, and m2 = 3 partial regions 52 (52_c, 52_d, 52_e) are arranged in the second row.

[0071] In Fig. 19, n = 2, m1 = 1, m2 = 4, and the partition direction is horizontal. That is, the layout candidate in Fig. 19 is [1, 4: horizontal]. Because n = 2, the display unit 13 is divided into two rows. m1 = 1 partial region 52 (52_a) is arranged in the first row, and m2 = 4 partial regions 52 (52_b, 52_c, 52_d, 52_e) are arranged in the second row.

[0072] In Fig. 20, n = 1, m1 = 5, and the partition direction is horizontal. That is, the layout candidate in Fig. 20 is [5: horizontal]. Because n = 1, the display unit 13 is configured in one row and is not divided. m1 = 5 partial regions 52 (52_a, 52_b, 52_c, 52_d, 52_e) are arranged in the first row. This layout candidate is substantially the same layout as [1, 1, 1, 1, 1: vertical].

[0073] When the number of partial regions 52 is five, various patterns of arrangement candidates are possible, such as [5: horizontal], [5: vertical], [1,2,1,1: horizontal], [1,1,2,1: vertical], and [1,1,1,1: horizontal]. However, considering all patterns as arrangement candidates increases the amount of calculation required to select the optimal one from among them, which may result in a long processing time. Therefore, in this embodiment, multiple arrangement candidates are determined in advance for each number of partial regions 52. For example, when the number of partial regions 52 is five, only eight patterns may be predetermined as arrangement candidates: [1,4: horizontal], [1,4: vertical], [2,3: horizontal], [2,3: vertical], [1,2,2: horizontal], [1,2,2: vertical], [1,1,1,1,1: horizontal], and [1,1,1,1,1: vertical]. Patterns with different partition orientations for horizontal and vertical orientations may also be predetermined as arrangement candidates.

[0074] Fig. 21 is a diagram showing specific examples of shape patterns of provided moving images. For example, when provided moving images are provided by five users, the aspect ratios of the provided moving images may differ. In the example of Fig. 21, provided moving images 60_1 to 60_5 provided by the first to fifth users, respectively, all have different aspect ratios. In Fig. 21, the aspect ratio (width / height) values ​​increase from the top to the bottom.

[0075] The display control unit 181 may determine which arrangement candidate to adopt when determining the size and display position of the partial region 52. In this case, the display control unit 181 first obtains the aspect ratio of each provided moving image 60 to be displayed and sorts the images by the aspect ratio. Next, the display control unit 181 performs the following process for all arrangement candidates. First, the display control unit 181 selects an arrangement candidate to be processed (hereinafter referred to as a "processing target arrangement candidate") and sorts the partial regions 52 in the processing target arrangement candidate by the aspect ratio (W1 / H1 in FIG. 15 ) of the image display candidate region 55. The display control unit 181 allocates provided moving images 60 with larger aspect ratios to the partial regions 52 in descending order of the aspect ratio of the image display candidate region 55.

[0076] At this time, the image display area 56 is determined while maintaining the aspect ratio of the provided video 60 so that the area of ​​the image display area 56 in each image display candidate area 55 is maximized. Information representing the size and display position of the partial areas 52, the allocation of the provided video to each partial area 52, and the size and display position of the image display area 56 in each partial area 52 is referred to as arrangement candidate information. The display control unit 181 calculates the sum of the areas of the image display areas 56 in each partial area 52 in the processing target arrangement candidate (hereinafter referred to as the "total display area"). The display control unit 181 calculates the total display area for all arrangement candidates. The display control unit 181 selects the arrangement candidate that maximizes the total display area, and performs display according to the arrangement candidate information of the selected arrangement candidate.

[0077] 22 to 24 show specific examples of placement candidate information. FIG. 22 shows a specific example of placement candidate information in which the provided images shown in FIG. 21 are placed in the placement candidate shown in FIG. 16. The provided video 60_1 with the largest aspect ratio is assigned to the partial region 52_a having the image display candidate region 55 with the largest aspect ratio. The image display region 56_1 of the partial region 52_a is then determined so as to have the largest area while maintaining the aspect ratio of the provided video 60_1. For example, in the example of FIG. 22, the image display region 56_1 may be determined so that the height of the provided video 60_1 matches the height of the image display candidate region 55 of the partial region 52_a. In the example of FIG. 22, the image display candidate regions 55 of the remaining four partial regions 52 have the same aspect ratio. In this case, the provided videos 60 may be assigned to the partial regions 52 in descending order of aspect ratio, for example, based on the placement position. For example, partial regions 52 may be allocated in order from the highest level, or from the leftmost level within the same level. Therefore, in the example of FIG. 22 , partial regions 52_b, 52_c, 52_d, and 52_e are allocated in order of aspect ratio. The image display region 56 of each partial region 52 is determined so as to have the largest area while maintaining the aspect ratio of the allocated provided video 60. For example, in the example of FIG. 22 , the image display region 56 may be determined so that the height of each of the remaining four provided videos 60 matches the height of each image display candidate region 55.

[0078] FIG. 23 shows a specific example of placement candidate information in which the provided images shown in FIG. 21 are placed in the placement candidate shown in FIG. 17 . The provided video 60_1, which has the largest aspect ratio, is assigned to the partial region 52 having the image display candidate region 55 with the largest aspect ratio. However, in the example of FIG. 23 , the aspect ratios of the image display candidate regions 55 of the four partial regions 52 (52_b, 52_c, 52_d, and 52_e) are the same. In this case, the provided videos 60 may be assigned to the partial regions 52 in descending order of aspect ratio based on their placement positions, for example. For example, the provided videos 60 may be assigned to the partial regions 52 located in the rightmost column first, or in descending order within the same column. Therefore, in the example of FIG. 23 , the provided videos 60 are assigned to the partial regions 52_e, 52_d, 52_c, and 52_b in descending order of aspect ratio. The image display area 56 of each partial area 52 is determined so as to have the largest area while maintaining the aspect ratio of the assigned provided video 60. For example, in the example of Fig. 23, the image display area 56 may be determined so that the height of each provided video 60 matches the width of each image display candidate area 55.

[0079] FIG. 24 shows a specific example of placement candidate information in which the provided images shown in FIG. 21 are placed in the placement candidate shown in FIG. 18 . The provided video 60_1, which has the largest aspect ratio, is assigned to the partial region 52 having the image display candidate region 55 with the largest aspect ratio. However, in the example of FIG. 24 , the aspect ratios of the image display candidate regions 55 of the two partial regions 52 (52_a, 52_b) are the same. In this case, the provided videos 60 may be assigned to the partial regions 52 in descending order of aspect ratio based on their placement positions, for example. For example, the provided videos 60 may be assigned to the partial regions 52 in descending order of aspect ratio, starting with the partial region 52 located higher up the hierarchy, or, if they are in the same hierarchy, starting with the partial region 52 located further left. Therefore, in the example of FIG. 24 , the provided videos 60 are assigned to the partial regions 52_a, 52_b, 52_c, 52_b, and 52_e in descending order of aspect ratio. The image display area 56 of each partial area 52 is determined so as to have the largest area while maintaining the aspect ratio of the assigned provided video 60. For example, in the example of Fig. 24, the image display area 56 may be determined so that the width of each provided video 60 matches the width of each image display candidate area 55.

[0080] The display control unit 181 also acquires placement candidate information for placement candidates other than those shown in Figures 22 to 24. Then, the display control unit 181 selects the placement candidate that has the largest total display area from among the placement candidates. For example, the placement candidate information for the placement candidate shown in Figure 22 may be selected. Then, the display control unit 181 controls the display based on the selected placement candidate information.

[0081] The display control unit 181 may select and control the display of the above-described placement candidate information at a predetermined timing. For example, the display control may be performed at a predetermined cycle (e.g., every second, every 10 seconds, or every minute). The display control unit 181 may also perform the display control when a user of the device (user terminal 10) performs a predetermined operation. By performing the display control at such timing, it is possible to prevent the placement candidate information on the screen from changing at a timing unintended by the user. In other words, it is possible to prevent the display position or size of the provided image from changing at a timing unintended by the user. Specific examples of such predetermined operations include an operation to change the number of displayed partial areas 52, an operation to change the display position or size of the partial areas 52, and an operation to change the display area (e.g., a window) of an application (e.g., a web browser) of the conversation control system 100.

[0082] The display control unit 181 may execute the above-described display control at the timing when a user of another user terminal 10 performs a predetermined operation. Executing the display control at such timing can prevent the screen layout candidate information from changing unnecessarily frequently. In other words, it can prevent the display position of the provided image or the size of the display area from changing unnecessarily frequently. Specific examples of such predetermined operations include, for example, an operation to change the size of the provided video, an operation to start providing the provided video, and an operation to stop providing the provided video.

[0083] The display control of the reduced partial area 51 and partial area 52 shown in Figures 9 to 24 may be performed by the display control unit 181 of the user terminal 10, or by the display information generation unit 233 of the conversation control device 20.

[0084] The conversation control system 100 configured in this manner makes it possible to realize a screen display that is more convenient for the user when conversing with one or more other users. Specifically, this is as follows: In the conversation control system 100, the size and position of the partial area 52 in which the video images provided by each user (provided video images) are displayed are determined so that the area in which the provided video images are displayed is larger. This makes it possible to realize a screen display that is more convenient for the user.

[0085] Furthermore, in the conversation control system 100, the layout options to be considered are limited in advance when determining the size and position of the partial area 52 in which the video images (provided video images) provided by each user are displayed, which makes it possible to determine the display more quickly with less calculation.

[0086] In addition, the data of the provided video may be transmitted from the user terminal 10 that provides it (e.g., the source of sharing) to the user terminal 10 that receives it via the conversation control device 20, or it may be transmitted from the user terminal 10 that provides it to the user terminal 10 that receives it without passing through the conversation control device 20.

[0087] The conversation control unit 183 or the conversation control unit 234 may perform control so that the voice of the user displayed in the partial area 52 is output louder than the voice of the user displayed in the reduced partial area 51.

[0088] 25 is a diagram showing a specific example of arrangement candidate information according to a modified example. In FIG. 25, the width of the partial area 52 in the first row is shorter than the actual display width of the screen in the first row. In this way, the height and width of the partial area 52 do not necessarily have to be the maximum possible value within the screen or window area of ​​the display unit 13, and may be configured to be values ​​smaller than the maximum value. Furthermore, the sizes of multiple partial areas 52 arranged in the same row or column may be different from one another.

[0089] FIG. 26 is a diagram illustrating an outline of an example hardware configuration of an information processing device 900 applied to this embodiment. The information processing device 900 includes a processor 901, a main memory device 902, a communication interface 903, an auxiliary memory device 904, an input / output interface 905, and an internal bus 906. The processor 901, the main memory device 902, the communication interface 903, the auxiliary memory device 904, and the input / output interface 905 are communicably connected to each other via the internal bus 906. The information processing device 900 may be applied to, for example, a user terminal 10 and a conversation control device 20. In this case, for example, the communication units 11 and 21 may be configured using the communication interface 903. For example, the memory units 17 and 22 may be configured using the auxiliary memory device 904. Furthermore, the control units 18 and 23 may be configured using the processor 901 and the main memory device 902.

[0090] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0091] The present invention can be applied to technology for receiving and displaying multiple video images online.

[0092] REFERENCE SIGNS LIST 100... conversation control system, 10... user terminal, 20... conversation control device, 11... communication unit, 12... operation unit, 13... display unit, 14... audio input unit, 15... audio output unit, 16... imaging unit, 17... storage unit, 171... user information storage unit, 18... control unit, 181... display control unit, 182... conference control unit, 183... conversation control unit, 21... communication unit, 22... storage unit, 221... conference room information storage unit, 23... control unit, 231... user control unit, 232... conference room control unit, 233... display information generation unit, 234... conversation control unit, 30... conference room area image, 31... conversation area, 311... conference room name, 312... entry button, 313... details button, 32... listening area, 41... icon image, 91... outdoor area

Claims

1. A control system comprising: a communication unit that receives data of multiple moving images provided from another device; and a control unit that displays the moving images received by the communication unit by allocating them to multiple partial areas obtained by dividing a display screen into multiple parts, wherein the control unit determines the allocation of the partial areas to the moving images so as to maximize the sum of the areas displaying the multiple moving images.

2. A control system according to claim 1, wherein the control unit determines the allocation of the partial regions and the moving images based on the aspect ratio of each moving image and the aspect ratio of each partial region.

3. The control system of claim 2, wherein the control unit determines the allocation of the partial regions to the video images so that the order of the aspect ratios of each video image matches the order of the aspect ratios of each partial region.

4. A control method in which a computer receives data of a plurality of moving images provided from another device, the computer displays the received moving images by allocating them to a plurality of partial areas obtained by dividing a display screen into a plurality of areas, and the computer determines the allocation of the partial areas to the moving images so as to maximize the sum of the areas displaying the plurality of moving images.

5. A computer program for operating a computer as a control system comprising: a communication unit that receives data on a plurality of moving images provided from another device; and a control unit that displays the moving images received by the communication unit by allocating them to a plurality of partial areas obtained by dividing a display screen into a plurality of areas, wherein the control unit determines the allocation of the partial areas to the moving images so as to maximize the sum of the areas displaying the plurality of moving images.

Images