Image processing system, image processing method, and image processing program

Abstract

An image processing system according to one embodiment comprises a plurality of image processing servers that each manage a plurality of user terminals, and a central image processing server that is connected to the plurality of image processing servers. Each of the plurality of image processing servers comprises an image reception unit that receives image information including moving images that were captured by the plurality of user terminals, a score calculation unit that uses a prescribed algorithm to calculate a score for each of the received items of image information, and generates, in order from the highest score, a list including a prescribed number of items of user information of the user terminals, and a list transmission unit that transmits the list. The central image processing server comprises a list reception unit that receives the lists from the plurality of image processing servers, a user selection unit that selects, from the received lists, user information to be used for creating an image to be displayed on each of the plurality of user terminals, and a user information transmission unit that transmits the selected user information to each of the plurality of image processing servers.

Description

Technical Field

[0001] This invention relates to an image processing system, an image processing method, and an image processing program.

Background Art

[0002] In order to create a sense of unity between people gathered at a venue for an event or the like and people participating remotely from a location such as home, the images of people participating remotely are displayed on a screen or the like installed within the venue, or the videos of a plurality of people participating remotely are displayed on a screen possessed by a user participating remotely from another location.

[0003] For example, Patent Document 1 discloses a technology for reducing the composite processing load by resizing videos transmitted from a plurality of communication terminals and then creating a composite video in a multi-site conference system for holding a conference using a plurality of communication terminals located at multiple sites. Also, for example, Patent Document 2 discloses a technology for reducing unnecessary traffic on a network by selecting a value different from that of the transmission-side communication terminal for encoding parameters such as the bit rate or frame rate of videos transmitted from a plurality of communication terminals and encoding the composite video. Furthermore, Patent Document 3 discloses a technology for generating a composite video based on video arrangement information by magnifying each of the input videos by a plurality of magnification factors, generating a plurality of magnified input videos for each input video.

[0004] Also, Non-Patent Document 1 discloses the results of an investigation of the influence of video delay in an online dance participated by a plurality of people on the sense of unity, ease of dancing, or ease of synchronizing operation timing.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

[0006] [Non-Patent Document 1] Information Processing Society of Japan Research Report Vol.2022-DCC-31 No.2 "A Study on the Impact of Video Latency in Online Dance with Multiple Participants" [Overview of the project] [Problems that the invention aims to solve]

[0007] The number of terminals expected in multi-point conferencing is at most 50 or less. For events with many more users accessing the event than this, the number is too large, making it impossible to implement with conventional technologies.

[0008] Furthermore, in events with a small number of remotely connected users, the methods disclosed in the prior art can be applied, allowing all users' images to be combined and displayed on the same screen. However, since the images of all participating users are displayed regardless of their status, the display area per user becomes smaller. Therefore, the methods disclosed in the prior art do not necessarily lead to an improvement in the sense of unity among participants.

[0009] To solve these problems, it would be effective to prioritize displaying content based on the user's level of engagement. However, to achieve this, it is necessary to receive images from all users and then select and display content based on user input. As a result, the amount of data from images and input information becomes enormous, making the user selection process heavy and leading to significant delays in display.

[0010] Furthermore, when selecting users and displaying the combined video on a remote user's screen, it is necessary to send the images from the server creating the combined video to each user. In other words, the images of each selected user are first collected from each user to the video combining server, and after the combined video is created, the video is transferred to each user. This sequence of sending and receiving video creates redundancy in the path, which can cause delays.

[0011] This invention was made in view of the above circumstances, and its objective is to provide a technology that can reduce the processing load on each server by providing a video processing server and a central video processing server. Furthermore, its objective is to provide a technology that can make the distribution path of the synthesized video the shortest path for each video processing server and central video processing server. [Means for solving the problem]

[0012] To solve the above problems, one aspect of the present invention provides an image processing system comprising: a plurality of image processing servers, each managing a plurality of user terminals; and a central image processing server connected to the plurality of image processing servers, wherein each of the plurality of image processing servers comprises: an image receiving unit that receives image information including moving images captured by the plurality of user terminals; a storage unit that stores a predetermined algorithm; a score calculation unit that calculates a score for each of the received moving images using the predetermined algorithm and generates a list containing a predetermined number of user information for the user terminals in descending order of the scores; and a list transmission unit that transmits the list, wherein the central image processing server comprises: a list receiving unit that receives the list from the plurality of image processing servers; a user selection unit that selects user information from the list received from the plurality of image processing servers to be used to create images displayed on each of the plurality of user terminals; and a user information transmission unit that transmits the user information to each of the plurality of image processing servers. [Effects of the Invention]

[0013] According to one aspect of the present invention, by distributing the processing in each server, the processing speed can be increased and the overall delay can be reduced.

Brief Description of the Drawings

[0014] [Figure 1] FIG. 1 is a diagram showing an example of the outline of an image processing system according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the schematic configuration of the central image processing server. [Figure 3] FIG. 3 is a block diagram showing an example of the schematic configuration of the image processing server. [Figure 4] FIG. 4 is a block diagram showing an example of the schematic configuration of the user terminal. [Figure 5] FIG. 5 is a block diagram showing an example of the schematic configuration of the image relay router. [Figure 6] FIG. 6 is a sequence diagram showing an example of the operation for acquiring a moving image from the user terminal and displaying the composite image on the user terminal 30 in the image processing system. [Figure 7] FIG. 7 is a diagram showing an example of the composite image. [Figure 8] FIG. 8 is a diagram showing an example of the outline of an image processing system according to the second embodiment. [Figure 9] FIG. 9 is a block diagram showing an example of the schematic configuration of the score relay server 50. [Figure 10] FIG. 10 is a sequence diagram showing an example of the operation for acquiring a moving image from the user terminal and displaying the composite image on the user terminal in the image processing system. [Figure 11] FIG. 11 is a diagram showing an example of the outline of an image processing system according to a modification of the second embodiment. [Figure 12] FIG. 12 is a diagram showing an example of the composite image generated according to the instructions of the first central image processing server and the second central image processing server. [Figure 13]FIG. 13 is a sequence diagram showing an example of operations for acquiring a moving image from a user terminal and displaying a composite image on the user terminal in an image processing system. [Figure 14] FIG. 14 is a diagram showing an example of a first composite image and a second composite image. [Figure 15] FIG. 15 is a diagram showing an example of a composite image used in a modified example of the composite image. [Figure 16] FIG. 16 is a diagram showing an example of a background image used in a modified example of the composite image.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, embodiments of this invention will be described with reference to the drawings. In the following, the same or similar elements as those already described are denoted by the same or similar reference numerals, and duplicate explanations are basically omitted. For example, when there are a plurality of identical or similar elements, a common reference numeral may be used to describe each element without distinction, or a branch number may be used in addition to the common reference numeral to describe each element separately.

[0016] [First Embodiment] (Configuration) FIG. 1 is a diagram showing an example of the outline of an image processing system according to the first embodiment. In FIG. 1, the image processing system includes a central image processing server 10, a first image processing server 201, a second image processing server 202, a third image processing server 203, a first group user terminal 3a, a second group user terminal 3b, a third group user terminal 3c, a first image relay router 401, a second image relay router 402, and a third image relay router 403.

[0017] The first group of user terminals 3a consists of multiple user terminals: the first user terminal 301, the second user terminal 302, ..., and the nth user terminal 30n. The second group of user terminals 3b consists of multiple user terminals: the (n+1)th user terminal 30n+1, the (n+2)th user terminal 30n+2, ..., and the 2nth user terminal 302n. Similarly, the third group of user terminals 3c consists of multiple user terminals: the 2n+1st user terminal 302n+1, the 2n+2nd user terminal 302n+2, ..., and the 3nth user terminal 303n. Here, n is a positive integer, and for example, n = 50. Furthermore, while Figure 1 shows an example where the first group user terminal 3a, the second group user terminal 3b, and the third group user terminal 3c are composed of the same number of user terminals, it goes without saying that each group user terminal may be composed of a different number of user terminals.

[0018] In the following description, if it is not necessary to distinguish between the first image processing server 201, the second image processing server 202, and the third image processing server 203, they will simply be referred to as image processing server 20. If it is not necessary to distinguish between the first group user terminal 3a, the second group user terminal 3b, and the third group user terminal 3c, they will simply be referred to as group user terminal 3. Furthermore, if it is not necessary to distinguish between each user terminal, they will simply be referred to as user terminal 30. Similarly, if it is not necessary to distinguish between the first image relay router 401, the second image relay router 402, and the third image relay router 403, they will simply be referred to as image relay router 40.

[0019] The central image processing server 10 is connected to the first image processing server 201, the second image processing server 202, the third image processing server 203, and the second image relay router 402. The central image processing server 10 is configured as a general-purpose computer capable of receiving various information from the image processing servers 20 and transmitting necessary information to the image relay router 40 (in the example in Figure 1, the second image relay router 402).

[0020] The image processing server 20 is connected to the central image processing server 10, the user terminals 30 belonging to the group user terminals managed by the image processing server 20, and the image relay router 40. For example, the first image processing server 201 is connected to the first user terminal 301, the second user terminal 302, ..., and the nth user terminal 30n, all belonging to the first group user terminal 3a. Here, at least one of the image processing servers 20 may also serve as the central image processing server 10. In this case, it is not necessary to deploy the central image processing server 10.

[0021] The user terminal 30 is a user terminal 30 owned by a user who is participating remotely in an event or multi-point conference. The user terminal 30 may be a mobile device such as a smartphone or tablet, or a stationary device such as a personal computer. For example, the user terminal 30 may be a general-purpose computer equipped with a camera capable of capturing the user's face or entire body and transmitting the captured images to the image processing server 20.

[0022] The image relay router 40 is configured as a general-purpose computer capable of transmitting information received from the central image processing server 10 to other image relay routers 40 or image processing servers 20.

[0023] Figure 2 is a block diagram showing an example of the schematic configuration of the central image processing server 10. As shown in Figure 2, the central image processing server 10 comprises a control unit 11, a program storage unit 12, a data storage unit 13, a communication interface 14, and an input / output interface 15. The control unit 11, program storage unit 12, data storage unit 13, communication interface 14, and input / output interface 15 are connected to each other via a bus so as to be able to communicate with each other. Furthermore, the communication interface 14 may be connected so as to be able to receive various information from the image processing server 20 via a network and transmit various information to the image relay router 40. In addition, the input / output interface 15 is connected to an input device 16 and an output device 17 so as to be able to communicate with each other.

[0024] The control unit 11 controls the central image processing server 10. The control unit 11 includes a hardware processor such as a central processing unit (CPU). For example, the control unit 11 may be an integrated circuit capable of executing various programs.

[0025] The program storage unit 12 can use a combination of non-volatile memory that allows writing and reading at any time, such as EPROM (Erasable Programmable Read Only Memory), HDD (Hard Disk Drive), and SSD (Solid State Drive), as a storage medium, and non-volatile memory such as ROM (Read Only Memory). The program storage unit 12 stores programs necessary to execute various processes. In other words, the control unit 11 can perform various controls and operations by reading and executing programs stored in the program storage unit 12.

[0026] The data storage unit 13 is a storage device that uses a combination of non-volatile memory, such as an HDD or memory card, which allows for writing and reading at any time, and volatile memory, such as RAM (Random Access Memory), as storage media. The data storage unit 13 is used to store data acquired and generated during the process in which the control unit 11 executes a program and performs various processing.

[0027] The communication interface 14 includes one or more wired or wireless communication modules. For example, the communication interface 14 includes a communication module that connects to the image processing server 20 and the image relay router 40 via a network, either by wire or wirelessly. In other words, the communication interface 14 can be any general communication interface that can communicate with the image processing server 20 and the image relay router 40 under the control of the control unit 11 and send and receive various types of information.

[0028] The input / output interface 15 is connected to the input device 16 and the output device 17, etc. The input / output interface 15 is an interface that enables the transmission and reception of information between the input device 16 and the output device 17. The input / output interface 15 may be integrated with the communication interface 14. For example, the central image processing server 10 and at least one of the input device 16 or the output device 17 may be wirelessly connected using short-range wireless technology, and information may be transmitted and received using said short-range wireless technology.

[0029] The input device 16 may include, for example, a keyboard or pointing device for an administrator managing the central image processing server 10 to input various information to the central image processing server 10. The input device 16 may also include a reader for reading data to be stored in the program storage unit 12 or data storage unit 13 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.

[0030] The output device 17 includes a display or the like that shows various information received from the image processing server 20.

[0031] Next, the software configuration of the central image processing server 10 in the embodiment will be described in more detail. The control unit 11 comprises a receiving control unit 111, a user selection unit 112, and a transmitting control unit 113.

[0032] The reception control unit 111 functions as a list receiving unit that receives list information from each of the image processing servers 20. Details of the list information will be described later. The reception control unit 111 then stores the received list information in the reception information storage unit 131, which will be described later.

[0033] The user selection unit 112 generates a predetermined list based on list information received from each of the multiple image processing servers. The user selection unit then generates user ID information, including user IDs included in the generated list. Of course, the information included in the user ID information is not limited to user IDs, but can be any information that can identify the user terminal. The user selection unit 112 stores the generated user ID information in the selection information storage unit 132. Details of the list generation method will be described later.

[0034] The transmission control unit 113 retrieves the user ID information stored in the selection information storage unit 132. Then, the transmission control unit 113 transmits the user ID information through the communication interface 14.

[0035] The data storage unit 13 includes a received information storage unit 131 and a selected information storage unit 132.

[0036] The received information storage unit 131 is used to store information received from external devices, including the image processing server 20, such as list information.

[0037] The selection information storage unit 132 is used to store the user ID information selected by the user selection unit 112.

[0038] Figure 3 is a block diagram showing an example of the schematic configuration of the image processing server 20. As shown in Figure 3, the image processing server 20 comprises a control unit 21, a program storage unit 22, a data storage unit 23, a communication interface 24, and an input / output interface 25. The control unit 21, program storage unit 22, data storage unit 23, communication interface 24, and input / output interface 25 are connected to each other via a bus so as to be able to communicate with each other. Furthermore, the communication interface 24 may be connected so as to be able to receive various information from a user terminal 30 via a network and transmit the received information to the central image processing server 10. The communication interface 24 may also be connected so as to be able to receive various information from an image relay router 40 and transmit the received information to the user terminal 30. In addition, the input / output interface 25 is connected to an input device 26 and an output device 27 so as to be able to communicate with each other.

[0039] The control unit 21 controls the image processing server 20. The control unit 21 includes a hardware processor such as a central processing unit (CPU). For example, the control unit 21 may be an integrated circuit capable of executing various programs.

[0040] The program storage unit 22 can use a combination of non-volatile memory, such as EPROM, HDD, or SSD, which allows for on-demand writing and reading, and other non-volatile memory, such as ROM, as storage media. The program storage unit 22 stores programs necessary for executing various processes. In other words, the control unit 21 can perform various controls and operations by reading and executing programs stored in the program storage unit 22.

[0041] The data storage unit 23 is a storage device that uses a combination of non-volatile memory, such as an HDD or memory card, which allows for writing and reading at any time, and volatile memory, such as RAM, as storage media. The data storage unit 23 is used to store data acquired and generated during the process in which the control unit 21 executes a program and performs various processing.

[0042] The communication interface 24 includes one or more wired or wireless communication modules. For example, the communication interface 24 includes communication modules that connect via a wired or wireless connection to the central image processing server 10, user terminals 30, and image relay router 40 over a network. The communication interface 24 may have separate interfaces for input during transmission, output during transmission, input during reception, output during reception, and the receiving side interface during reception. In other words, the communication interface 24 can be any general communication interface that can communicate with the central image processing server 10, user terminals 30, and image relay router 40 under the control of the control unit 21 and send and receive various types of information.

[0043] The input / output interface 25 is connected to the input device 26 and the output device 27, etc. The input / output interface 25 is an interface that enables the transmission and reception of information between the input device 26 and the output device 27. The input / output interface 25 may be integrated with the communication interface 24. For example, the image processing server 20 and at least one of the input device 26 or the output device 27 may be wirelessly connected using short-range wireless technology, and information may be transmitted and received using said short-range wireless technology.

[0044] The input device 26 may include, for example, a keyboard or pointing device for an administrator managing the image processing server 20 to input various information to the image processing server 20. The input device 26 may also include a reader for reading data to be stored in the program storage unit 22 or data storage unit 23 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.

[0045] The output device 27 includes a display or the like that shows various information received from the image processing server 20.

[0046] Next, the software configuration of the image processing server 20 in the embodiment will be described in more detail. The control unit 21 comprises a receiving control unit 211, a score calculation unit 212, a transmission control unit 213, and a composite image generation unit 214.

[0047] The receiving control unit 211 functions as an image receiving unit that receives image information from each of the user terminals 30 belonging to the group user terminal 3 via the communication interface 24. Details of the image information will be described later. The receiving control unit 211 stores the received image information in the received information storage unit 231. The receiving control unit 211 also functions as a user ID receiving unit that receives user ID information. Details of the user ID information will be described later. Furthermore, the receiving control unit 211 also functions as a video receiving unit that receives video of user terminals managed by other image processing servers specified by the user ID information.

[0048] The score calculation unit 212 calculates a score for the image information using a predetermined algorithm stored in the auxiliary information storage unit 232, which will be described later. The details of the predetermined algorithm will be described later.

[0049] The transmission control unit 213 functions as a list transmission unit that transmits list information to the central image processing server 10. Details of the list information will be described later. The transmission control unit 213 also functions as a video transmission unit that transmits user image information to the first image relay router 401. Details of the user image information will be described later. The transmission control unit 213 also functions as a user terminal transmission unit that transmits the composite image, described later, to the user terminal 30 managed by the image processing server.

[0050] The composite image generation unit 214 generates a composite image by combining video images of user IDs included in the list information. Details of the video image combination method will be described later.

[0051] The data storage unit 23 includes a received information storage unit 231 and an auxiliary information storage unit 232.

[0052] The received information storage unit 231 is used to store information received by the receiving control unit 211, such as image information received from the user terminal 30.

[0053] The auxiliary information storage unit 232 is used to store various types of information used to calculate the score described later. For example, the auxiliary information storage unit 232 stores a predetermined algorithm. For example, if the predetermined algorithm is an algorithm for calculating a score based on a predetermined action, the auxiliary information storage unit 232 may store trained video footage that has been learned from multiple video footages of a person performing a predetermined movement.

[0054] Figure 4 is a block diagram showing an example of the schematic configuration of the user terminal 30. As shown in Figure 4, the user terminal 30 comprises a control unit 31, a program storage unit 32, a data storage unit 33, a communication interface 34, and an input / output interface 35. The control unit 31, program storage unit 32, data storage unit 33, communication interface 34, and input / output interface 35 are connected to each other via a bus so as to be able to communicate with one another. Furthermore, the communication interface 34 may be connected to the image processing server 20 via a network so as to be able to send and receive various information. In addition, the input / output interface 35 is connected to the input device 36, the output device 37, and the camera 38 so as to be able to communicate with one another.

[0055] The control unit 31 controls the user terminal 30. The control unit 31 includes a hardware processor such as a central processing unit (CPU). For example, the control unit 31 may be an integrated circuit capable of executing various programs.

[0056] The program storage unit 32 can use a combination of non-volatile memory, such as EPROM, HDD, or SSD, which allows for on-demand writing and reading, and other non-volatile memory, such as ROM, as storage media. The program storage unit 32 stores programs necessary for executing various processes. In other words, the control unit 31 can perform various controls and operations by reading and executing programs stored in the program storage unit 32.

[0057] The data storage unit 33 is a storage device that uses a combination of non-volatile memory, such as an HDD or memory card, which allows for writing and reading at any time, and volatile memory, such as RAM, as storage media. The data storage unit 33 is used to store data acquired and generated during the process in which the control unit 31 executes a program and performs various processing.

[0058] The communication interface 34 includes one or more wired or wireless communication modules. For example, the communication interface 34 includes a communication module that connects to the image processing server 20 via a network, either wired or wirelessly. In other words, the communication interface 34 can be any general communication interface that can communicate with the image processing server 20 and send and receive various types of information under the control of the control unit 31.

[0059] The input / output interface 35 is connected to the input device 36, output device 37, camera 38, etc. The input / output interface 35 is an interface that enables the transmission and reception of information between the input device 36, output device 37, and camera 38. The input / output interface 35 may be integrated with the communication interface 34. For example, the user terminal 30 and at least one of the input device 36, output device 37, or camera 38 may be wirelessly connected using short-range wireless technology, and information may be transmitted and received using said short-range wireless technology.

[0060] The input device 36 may include, for example, a keyboard or pointing device for a user possessing the user terminal 30 to input various information into the user terminal 30. The input device 36 may also include a reader for reading data to be stored in the program storage unit 32 or data storage unit 33 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.

[0061] The output device 37 includes a display or the like that displays various information received from the user terminal 30.

[0062] Camera 38 is a general-purpose camera capable of capturing the face or entire body of a user participating in an event or multi-point conference. Camera 38 may also be integrated with the user terminal 30. Camera 38 outputs the captured image to the control unit 31 via the input / output interface 35.

[0063] Next, the software configuration of the user terminal 30 in the embodiment will be described in more detail. The control unit 31 comprises an image acquisition unit 311, a transmission control unit 312, a reception control unit 313, and an image control unit 314.

[0064] The image acquisition unit 311 acquires the captured image taken by the camera 38. Here, the captured image may be a video of a user holding the user terminal 30 waving a chemical light stick. The image acquisition unit 311 outputs the captured image to the transmission control unit 312. The image acquisition unit 311 may also store the captured image in the image information storage unit 331.

[0065] The transmission control unit 312 transmits the image information to the first image processing server 201. The transmission control unit 312 generates image information including the captured image received from the image acquisition unit 311 or stored in the image information storage unit 331 and the user ID of the user terminal 30. The transmission control unit 312 transmits the generated image information to the image processing server 20 via the communication interface 34.

[0066] The receiving control unit 313 receives the composite image. Details of the composite image will be described later. The receiving control unit 313 stores the received composite image in the receiving information storage unit 332.

[0067] The image control unit 314 controls the display of the composite image on the display of the output device 37 via the input / output interface 35.

[0068] The data storage unit 33 comprises an image information storage unit 331 and a received information storage unit 332. The image information storage unit 331 is used to store information acquired by the image acquisition unit 311, including captured images. The received information storage unit 332 is used to store information received by the reception control unit 313, including composite images.

[0069] Figure 5 is a block diagram showing an example of the schematic configuration of the image relay router 40. As shown in Figure 5, the image relay router 40 comprises a control unit 41, a program storage unit 42, a data storage unit 43, a communication interface 44, and an input / output interface 45. The control unit 41, program storage unit 42, data storage unit 43, communication interface 44, and input / output interface 45 are connected to each other via a bus so as to be able to communicate with one another. Furthermore, the communication interface 44 may be connected so as to be able to receive various information from the central image processing server 10 via the network and transmit it to other image relay routers 40 or image processing servers 20. In addition, the input / output interface 45 is connected to an input device 46 and an output device 47 so as to be able to communicate with one another.

[0070] The control unit 41 controls the image relay router 40. The control unit 41 includes a hardware processor such as a central processing unit (CPU). For example, the control unit 41 may be an integrated circuit capable of executing various programs.

[0071] The program storage unit 42 can use a combination of non-volatile memory, such as EPROM, HDD, or SSD, which allows for on-demand writing and reading, and other non-volatile memory, such as ROM, as storage media. The program storage unit 42 stores programs necessary for executing various processes. In other words, the control unit 41 can perform various controls and operations by reading and executing programs stored in the program storage unit 42.

[0072] The data storage unit 43 is a storage device that uses a combination of non-volatile memory, such as an HDD or memory card, which allows for writing and reading at any time, and volatile memory, such as RAM, as storage media. The data storage unit 43 is used to store data acquired and generated during the process in which the control unit 41 executes a program and performs various processing.

[0073] The communication interface 44 includes one or more wired or wireless communication modules. For example, the communication interface 44 includes communication modules that connect via wired or wireless connections to the central image processing server 10, image processing servers 20, and other image relay routers 40 over a network. In other words, the communication interface 44 can be any general communication interface that, under the control of the control unit 41, can communicate with the central image processing server 10, image processing servers 20, and other image relay routers 40 over a network and send and receive various types of information.

[0074] The input / output interface 45 is connected to the input device 46, the output device 47, etc. The input / output interface 45 is an interface that enables the transmission and reception of information between the input device 46 and the output device 47. The input / output interface 45 may be integrated with the communication interface 44. For example, the image relay router 40 and at least one of the input device 46 or the output device 47 may be wirelessly connected using short-range wireless technology, and information may be transmitted and received using said short-range wireless technology.

[0075] The input device 46 may include, for example, a keyboard or pointing device for an administrator managing the image relay router 40 to input various information to the image relay router 40. The input device 46 may also include a reader for reading data to be stored in the program storage unit 42 or data storage unit 43 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.

[0076] The output device 47 includes a display or the like that displays various information received from the image relay router 40.

[0077] Next, the software configuration of the image relay router 40 in the embodiment will be described in more detail. The control unit 41 comprises a receiving control unit 411 and a transmitting control unit 412.

[0078] The receiving control unit 411 receives user ID information or user image information through the communication interface 44 and outputs the received user ID information or user image information to the transmitting control unit 412. Details of the user ID information will be described later. The receiving control unit 411 may also store the user ID information and user image information in the received information storage unit 431.

[0079] The transmission control unit 412 transmits user ID information or user image information to another image relay router 40 or image processing server 20 via the communication interface 44. As described above, the receivability control unit 411 and the transmission control unit 412 operate as a user ID information relay unit that relays user ID information and as a video relay unit that relays user image information.

[0080] The data storage unit 43 includes a received information storage unit 431. The received information storage unit 431 is used to store information received by the receiving control unit 411, including user ID information and user image information.

[0081] (operation) Figure 6 is a sequence diagram showing an example of the operation in an image processing system to acquire moving images from a user terminal 30 and display a composite image on the user terminal 30. The sequence shown in Figure 6 is realized by the control unit 11 of the central image processing server 10, the control unit 21 of the image processing server 20, the control unit 31 of the user terminal 30, and the control unit 41 of the image relay router 40 reading and executing the programs stored in the program storage units 12, 22, 32, and 42, respectively.

[0082] This sequence is initiated, for example, when the user terminal 30 receives an instruction that the user will participate in an event or multi-point conference, or when the user terminal 30 senses that the start time of the event or multi-point conference has arrived.

[0083] In step ST101, the image acquisition unit 311 of each user terminal 30 belonging to the first group user terminal 3a acquires the captured image taken by the camera 38. Here, the captured image may be a video of the user holding the user terminal 30 waving a chemical light. The image acquisition unit 311 outputs the captured image to the transmission control unit 312. The image acquisition unit 311 may store the captured image in the image information storage unit 331.

[0084] In step ST102, the transmission control unit 312 transmits image information to the first image processing server 201. The transmission control unit 312 generates image information including the captured image received from the image acquisition unit 311 or stored in the image information storage unit 331 and the user ID of the user terminal 30, and transmits the generated image information to the first image processing server 201 via the communication interface 34. Here, the user ID can be any user information that uniquely identifies each user terminal 30, for example, assigning 1 to the first user terminal 301 and 2 to the second user terminal 302. In this embodiment, for simplicity, we will use user IDs for explanation, but of course, in this embodiment, user information may be used instead of user IDs. The first image processing server 201 will then receive captured images from n user terminals 30 belonging to the first group of user terminals 3a. Here, n may be, for example, 50. In other words, the first image processing server 201 will receive captured images from user terminals 30 to which user IDs 1 to 50 belonging to the first group user terminal 3a have been assigned.

[0085] In step ST103, the score calculation unit 212 of the first image processing server 201 calculates a score. The reception control unit 211 of the first image processing server 201 receives image information from each of the user terminals 30 belonging to the first group user terminal 3a via the communication interface 24. The reception control unit 211 stores the received image information in the reception information storage unit 231.

[0086] The score calculation unit 212 obtains the captured image and user ID included in the image information stored in the received information storage unit 231. Then, the score calculation unit 212 calculates a score for each captured image using a predetermined algorithm stored in the auxiliary information storage unit 232.

[0087] For example, the predetermined algorithm may be a score calculation algorithm based on a predetermined movement of a person. The auxiliary information storage unit 232 stores pre-trained video images that have been learned from multiple video images of a person performing a predetermined action under various backgrounds. Here, the predetermined action, in the example above, may be a person swinging a chemical light that emits a specific color in a predetermined manner. The score calculation unit 212 then calculates a score based on the degree of match between the pre-trained video images and the captured images.

[0088] Here, the score calculation unit 212 is not limited to the score calculation method described above. For example, the score calculation unit 212 may calculate the ratio of the number of pixels in a brightness or color difference range corresponding to a specific color to the total number of pixels in the captured image, and calculate a score based on whether the calculated ratio falls within a predetermined range.

[0089] Furthermore, the specified algorithm is not limited to those described above. For example, it could be an algorithm that converts audio contained in a video into text using speech-to-text technology and calculates a score based on the degree of match with a predetermined string. Alternatively, the specified algorithm could be an algorithm that calculates a score based on the volume of audio contained in a video.

[0090] Furthermore, the control unit 31 of the user device 30 may transmit image information including, instead of or in addition to, the mouse click position or operation information of an input device 36 such as a joystick (for example, the direction of movement of the joystick, the movement of the user's fingers on the display (for example, movement along a virtual chemical light displayed on the screen)). In this case, the algorithm may be an algorithm that calculates a score based on the degree of match between the received image information and specific mouse position or operation information of the input device 36 stored in advance in the auxiliary information storage unit 232.

[0091] In other words, the algorithm can be any algorithm that can calculate a score based on arbitrary image information sent from the user terminal 30 that the user has been involved with.

[0092] The score calculation unit 212 selects a predetermined number of users from the top-scoring users among those whose scores have been calculated. Here, the predetermined number can be any number, but in this case, it will be 4. For example, suppose the score calculation unit 212 has calculated pairs of the top 4 user IDs and their calculated scores as shown in List 1-1 below. That is, the score calculation unit 212 generates a list containing a predetermined number of user information for user terminals. The list may include the scores corresponding to the user information. The user information may be a user ID, or it may be any information that identifies the user terminal. The score calculation unit 212 outputs the list information, including List 1-1, to the transmission control unit 213.

[0093] List 1-1 User ID, score 1,80 2,70 10,90 15, 50

[0094] In step ST104, the transmission control unit 213 of the first image processing server 201 transmits the list information to the central image processing server 10 through the communication interface 24.

[0095] In step ST105, the image acquisition unit 311 of each user terminal 30 belonging to the second group of user terminals 3b acquires the captured image taken by the camera 38. Similar to step ST101, the image acquisition unit 311 outputs the captured image to the transmission control unit 312.

[0096] In step ST106, the transmission control unit 312 transmits the image information to the second image processing server 202. Similar to step ST102, the transmission control unit 312 transmits the image information to the second image processing server 202 via the communication interface 34. That is, the second image processing server 202 receives captured images from, for example, user terminals 30 to which user IDs 51 to 100 belonging to the second group user terminal 3b have been assigned.

[0097] In step ST107, the score calculation unit 212 of the second image processing server 202 calculates the score. Similar to step ST103, the score calculation unit 212 calculates the score for each user terminal 30 and selects a predetermined number of users from the top-scoring users. Here, for example, suppose the score calculation unit 212 calculates the top four user IDs and their calculated scores as shown in List 1-2 below. That is, the score calculation unit 212 generates a list containing a predetermined number of user terminal user information entries. The score calculation unit 212 outputs the list information, including List 1-2, to the transmission control unit 213.

[0098] List 1-2 User ID, score 51, 85 52, 75 65, 55 70, 45

[0099] In step ST108, the transmission control unit 213 of the second image processing server 202 transmits the list information to the central image processing server 10. Similar to step ST104, the transmission control unit 213 transmits the list information to the central image processing server 10 through the communication interface 34.

[0100] In step ST109, the image acquisition unit 311 of each user terminal 30 belonging to the third group user terminal 3c acquires the captured image taken by the camera 38. Similar to step ST101, the image acquisition unit 311 outputs the captured image to the transmission control unit 312.

[0101] In step ST110, the transmission control unit 312 transmits the image information to the third image processing server 203. Similar to step ST102, the transmission control unit 312 transmits the image information to the third image processing server 203 via the communication interface 34. That is, the third image processing server 203 receives captured images from, for example, user terminals 30 to which user IDs 101 to 150 belonging to the third group user terminal 3c have been assigned.

[0102] In step ST111, the score calculation unit 212 of the third image processing server 203 calculates a score. Similar to step ST103, the score calculation unit 212 calculates a score for each user terminal 30 and selects a predetermined number of users from the top-scoring users. Here, for example, suppose the score calculation unit 212 calculates the top four user IDs and their calculated scores as shown in List 1-3 below. That is, the score calculation unit 212 generates a list containing a predetermined number of user terminal user information entries. The score calculation unit 212 outputs the list information, including List 1-3, to the transmission control unit 213.

[0103] List 1-3 User ID, score 101, 83 102, 73 111, 53 120, 63

[0104] In step ST112, the transmission control unit 213 of the third image processing server 203 transmits the list information to the central image processing server 10. Similar to step ST104, the transmission control unit 213 transmits the list information to the central image processing server 10 through the communication interface 34.

[0105] In step ST113, the user selection unit 112 selects a predetermined number of users. The receiving control unit 111 receives list information from each of the image processing servers 20. The receiving control unit 111 stores the received list information in the receiving information storage unit 131. The user selection unit 112 selects a predetermined number of users from the list 1-1 to list 1-3 with the highest scores. That is, the user selection unit 112 selects user information from the list to be used to create a composite image to be displayed on each of the multiple user terminals. The predetermined number can be any number of multiple people, and here it is set to 4 people. Of the above lists 1-1 to list 1-3, the top 4 with the highest scores are user IDs 1, 10, 51, and 101. Therefore, the user selection unit 112 generates list 1-4.

[0106] List 1-4 User ID, score 1,80 10,90 51, 85 101, 83

[0107] In other words, the user selection unit 112 determines that users with user IDs 1, 10, 51, and 101 are the users that should be displayed on the screen. The user selection unit 112 then generates user ID information that includes user IDs 1, 10, 51, and 101 included in list 1-4 and outputs it to the transmission control unit 113. Here, the user selection unit 112 may configure the user ID information for the first image processing server 201 to include only user IDs 1 and 10, the user ID information for the second image processing server 202 to include only user ID 51, and the user ID information for the third image processing server 203 to include only user ID 101. In other words, the user ID information may include all of list 1-4, or it may be generated separately for each image processing server 20.

[0108] In step ST114, the transmission control unit 113 transmits user ID information to the second image relay router 402 via the communication interface 14.

[0109] In step ST115, the transmission control unit 412 of the second image relay router 402 transmits the user ID information to the first image relay router 401 and the third image relay router 403. The reception control unit 411 of the second image relay router 402 receives the user ID information through the communication interface 44 and outputs the received user ID information to the transmission control unit 412. The transmission control unit 412 then transmits the user ID information to the first image relay router 401 and the third image relay router 403.

[0110] In step ST116, the transmission control unit 412 of the first image relay router 401 transmits user ID information to the first image processing server 201 and the second image processing server 202. The reception control unit 411 of the first image relay router 401 receives the user ID information through the communication interface 44 and outputs the received user ID information to the transmission control unit 412. The transmission control unit 412 then transmits the user ID information to the first image processing server 201 and the second image processing server 202.

[0111] In step ST117, the transmission control unit 412 of the third image relay router 403 transmits user ID information to the third image processing server 203. Similarly to step ST117, the transmission control unit 412 of the third image relay router 403 transmits user ID information to the third image processing server 203.

[0112] In step ST118, the transmission control unit 213 of the first image processing server 201 transmits user image information to the first image relay router 401. The reception control unit 211 of the first image processing server 201 receives user ID information. The reception control unit 211 outputs the user ID information to the transmission control unit 213. Based on the information in list 1-4 included in the user ID information, the transmission control unit 213 knows that video images of user ID 1 and user ID 10 are needed from the user terminals 30 managed by the first image processing server 201. The transmission control unit 213 then retrieves video images corresponding to user ID 1 and user ID 10 from the reception information storage unit 231, generates user image information including the user IDs and video images, and transmits the generated user image information to the first image relay router 401.

[0113] In step ST119, the transmission control unit 213 of the second image processing server 202 transmits user image information to the first image relay router 401. Similar to step ST119, the transmission control unit 213 of the second image processing server 202 generates user image information including user ID 51 and a video corresponding to user ID 51, and transmits the generated user image information to the first image relay router 401.

[0114] In step ST120, the transmission control unit 213 of the third image processing server 203 transmits user image information to the third image relay router 403. Similar to step ST119, the transmission control unit 213 of the third image processing server 203 generates user image information including user ID 101 and a video corresponding to user ID 101, and transmits the generated user image information to the third image relay router 403.

[0115] In step ST121, the transmission control unit 412 of the second image relay router 402 transmits user image information to the first image relay router 401 and the third image relay router 403. The reception control unit 411 of the second image relay router 402 receives user image information from the first image processing server 201 and the second image processing server 202, and outputs the received user image information to the transmission control unit 412. The transmission control unit 412 transmits the user image information to the second image relay router 402. Similarly, the transmission control unit 412 of the third image relay router 403 transmits the user image information received from the third image processing server 203 to the second image relay router 402.

[0116] The receiving control unit 411 of the second image relay router 402 receives user image information from the first image relay router 401 and the third image relay router 403. The second image relay router 402 receives user image information transmitted by the first image processing server 201, the second image processing server 202, and the third image processing server 203. That is, the second image relay router 402 receives the video corresponding to the user ID in List 1-4. Then, the transmitting control unit 412 of the second image relay router 402 transmits the user image information received from the third image relay router 403 to the first image relay router 401. Furthermore, the transmitting control unit 412 of the second image relay router 402 transmits the user image information received from the first image relay router 401 to the third image relay router 403.

[0117] In step ST122, the transmission control unit 412 of the first image relay router 401 transmits user image information to the first image processing server 201 and the second image processing server 202. The reception control unit 411 of the first image relay router 401 receives the user image information transmitted by the third image processing server 203 via the second image relay router 402 and the third image relay router 403. As a result, the first image relay router 401 receives all the moving images corresponding to List 1-4. Then, the transmission control unit 412 of the first image relay router 401 transmits the user image information transmitted by the second image processing server 202 and the third image processing server 203 to the first image processing server 201. Similarly, the transmission control unit 412 transmits the user image information transmitted by the first image processing server 201 and the third image processing server 203 to the second image processing server 202.

[0118] In step ST123, the transmission control unit 412 of the third image relay router 403 transmits user image information to the third image processing server 203. The reception control unit 411 of the third image relay router 403 receives the user image information transmitted by the first image processing server 201 and the second image processing server 202 via the first image relay router 401 and the second image relay router 402. As a result, the third image relay router 403 receives all the moving images corresponding to List 1-4. Then, the transmission control unit 412 of the third image relay router 403 transmits the user image information transmitted by the first image processing server 201 and the second image processing server 202 to the third image processing server 203.

[0119] Here, in steps ST118 to ST123, the transmission control unit 213 of the image processing server 20 may transmit user image information to other image processing servers 20 via multicast through the image relay router 40. That is, the reception control unit 211 of the image processing server 20 will individually receive, via multicast, video images of high-scoring users (video images corresponding to List 1-4) that it does not have stored in its memory from other image processing servers 20.

[0120] The transmission control unit 213 may reduce the size of the video image before transmitting it to the other image processing server 20. Here, the reduction method includes not only reducing the resolution, but also any method that reduces the size from the original video image, such as reducing the number of pixels, compressing the image, and methods that combine these.

[0121] Multicast is standardized by the IETF (Internet Engineering Task Force) as follows: RFC 1112: Host Extensions for IP Multicasting RFC 2236: Internet Group Management Protocol RFC 3973: Protocol Independent Multicast - Dense Mode RFC 4607: Source-Specific Multicast for IP RFC 7761: Protocol Independent Multicast - Sparse Mode (PIM-SM)

[0122] In step ST124, the composite image generation unit 214 of the first image processing server 201 generates a composite image. The receiving control unit 211 of the first image processing server 201 receives user image information and stores the received user image information in the receiving information storage unit 231. The composite image generation unit 214 generates a composite image by combining the video images of the user IDs corresponding to List 1-4.

[0123] If the video included in the received user image information is reduced in size, the composite image generation unit 214 may reduce the video stored in the received information storage unit 231, i.e., the video received from the user terminal 3 managed by the first image processing server 201, to match the reduced video size, and then generate a composite image. Alternatively, if the video included in the received user image information is compressed, the composite image generation unit 214 may decompress the received video before generating a composite image.

[0124] Figure 7 shows an example of a composite image. As shown in Figure 7, the composite image generation unit 214 generates a composite image by combining video footage corresponding to user ID 1, user ID 10, user ID 51, and user ID 101. These video footage may be combined using timestamps included in the video footage to ensure they are from the same time. This ensures that the composite image generated by the composite image generation unit 214 consists of video footage that is temporally synchronized.

[0125] In step ST125, the transmission control unit 213 of the first image processing server 201 transmits the composite image to the user terminal 30 belonging to the first group user terminal 3a. The composite image generation unit 214 outputs the composite image to the transmission control unit 213. The transmission control unit 213 transmits the composite image to the first user terminal 301 to the 50th user terminal 3050 (the nth user terminal 30n) belonging to the first group user terminal 3a via the communication interface 24. The reception control unit 313 of each user terminal receives the composite image and stores it in the reception information storage unit 332. The image control unit 314 then controls the display of the output device 37 to display the composite image stored in the reception information storage unit 332. As a result, the user of the user terminal 30 belonging to the first group user terminal 3a can view the composite image.

[0126] In step ST126, the composite image generation unit 214 of the second image processing server 202 generates a composite image. Similar to the processing in step ST124, the composite image generation unit 214 of the second image processing server 202 generates a composite image by combining the video images of the user IDs corresponding to List 1-4.

[0127] If the video included in the received user image information is reduced in size, the composite image generation unit 214 may reduce the video stored in the received information storage unit 231, i.e., the video received from the user terminal 3 managed by the second image processing server 202, to match the reduced video size, and then generate a composite image. Alternatively, if the video included in the received user image information is compressed, the composite image generation unit 214 may decompress the received video before generating a composite image.

[0128] In step ST127, the transmission control unit 213 of the second image processing server 202 transmits the composite image to the user terminal 30 belonging to the second group user terminal 3b. Similar to step ST125, the transmission control unit 213 of the second image processing server 202 transmits the composite image to the 51st user terminal 3051 (the n+1st user terminal 30n+1) to the 100th user terminal 30100 (the 2nth user terminal 302n) belonging to the second group user terminal 3b via the communication interface 24. Similar to step ST125, the image control unit 314 of each user terminal controls the display of the output device 37 to display the composite image. As a result, the user of the user terminal 30 belonging to the second group user terminal 3b can view the composite image.

[0129] In step ST128, the composite image generation unit 214 of the third image processing server 203 generates a composite image. Similar to the processing in step ST124, the composite image generation unit 214 of the third image processing server 203 generates a composite image by combining the video images of the user IDs corresponding to List 1-4.

[0130] If the video included in the received user image information is reduced in size, the composite image generation unit 214 may reduce the video stored in the received information storage unit 231, i.e., the video received from the user terminal 3 managed by the third image processing server 201, to match the reduced video size, and then generate a composite image. Alternatively, if the video included in the received user image information is compressed, the composite image generation unit 214 may decompress the received video before generating a composite image.

[0131] In step ST129, the transmission control unit 213 of the third image processing server 203 transmits the composite image to the user terminal 30 belonging to the third group user terminal 3c. Similar to step ST125, the transmission control unit 213 of the third image processing server 203 transmits the composite image to the 101st user terminal 30101 (the 2n+1st user terminal 302n+1) to the 150th user terminal 30150 (the 3nth user terminal 303n) belonging to the third group user terminal 3c via the communication interface 24. Similar to step ST125, the image control unit 314 of each user terminal controls the display of the output device 37 to display the composite image. As a result, the user of the user terminal 30 belonging to the third group user terminal 3c can view the composite image.

[0132] Here, in step ST114 of the process described above, the transmission control unit 113 of the central image processing server 10 may transmit the user ID to the image processing server 20 without going through the image relay router 40. In this case, steps ST115 to ST117 can be omitted.

[0133] Furthermore, in steps ST118, ST119, and ST120 of the above-described process, the transmission control units 213 of the first image processing server 201, the second image processing server 202, and the third image processing server 203 may transmit user image information to their respective image processing servers 20 via multicast without going through the image relay router 40.

[0134] As described above, if the central image processing server 10 directly sends the user ID to the image processing server 20, and the image processing server 20 directly sends the user image information, then the image relay router 40 can obviously be omitted.

[0135] Alternatively, the image processing server 20 may be divided into a receiving image processing server 20 and a transmitting image processing server 20. In this case, the receiving image processing server 20 will be responsible for steps ST103 to ST104, ST107 to ST108, ST111 to ST112, and ST118 to ST120. The transmitting image processing server 20 will be responsible for steps ST124 to ST129. Furthermore, by dividing the image processing server 20 into a receiving image processing server 20 and a transmitting image processing server 20, the communication interface 24 does not need to have an input interface for transmission, an output interface for transmission, an input interface for reception, an output interface for reception, and a receiving-side interface for reception.

[0136] By separating the receiving image processing server 20 and the transmitting image processing server 20 in this way, processing can be further distributed, reducing the processing load on each server.

[0137] (Effects of the first embodiment) According to the first embodiment, by arranging the central image processing server 10 and the image processing server 20, it is possible to reduce the processing load on each server.

[0138] Furthermore, by having the image processing server 20 transmit user image information directly or via the image relay router 40, it becomes possible to send and receive user image information for creating a composite image via the shortest path, thereby reducing the delay when creating the composite image.

[0139] [Second Embodiment] (composition) Figure 8 is a diagram showing an example of an overview of the image processing system according to the second embodiment. As shown in Figure 8, the second embodiment differs from the first embodiment in that, in addition to the devices described in the first embodiment, a first score relay server 501 and a second score relay server 502 are also provided.

[0140] Here, if there is no need to distinguish between the first score relay server 501 and the second score server, they are simply referred to as score relay server 50. Also, although Figure 8 shows only the first score relay server 501 and the second score relay server 502, it goes without saying that any number of score relay servers 50 can be deployed.

[0141] The first score relay server 501 is connected to the central image processing server 10, the first image processing server 201, and the second image processing server 202. Furthermore, the second score relay server 502 is connected to the central image processing server 10 and the third image processing server.

[0142] The score relay server 50 receives list information from the image processing server 20 and transmits the received list information to the central image processing server 10. Furthermore, if the score relay server 50 receives list information from multiple image processing servers 20, it generates new list information and transmits the new list information to the central image processing server 10.

[0143] Figure 9 is a block diagram showing an example of the schematic configuration of the score relay server 50. As shown in Figure 9, the score relay server 50 comprises a control unit 51, a program storage unit 52, a data storage unit 53, a communication interface 54, and an input / output interface 55. The control unit 51, program storage unit 52, data storage unit 53, communication interface 54, and input / output interface 55 are connected to each other via a bus so that they can communicate with one another. Furthermore, the communication interface 54 may be connected so that it can receive various information from the image processing server 20 via a network and transmit the received information to the central image processing server 10.

[0144] The control unit 51 controls the score relay server 50. The control unit 51 includes a hardware processor such as a central processing unit (CPU). For example, the control unit 51 may be an integrated circuit capable of executing various programs.

[0145] The program storage unit 52 can use a combination of non-volatile memory, such as EPROM, HDD, or SSD, which allows for on-demand writing and reading, and other non-volatile memory such as ROM, as storage media. The program storage unit 52 stores programs necessary for executing various processes. In other words, the control unit 51 can perform various controls and operations by reading and executing programs stored in the program storage unit 52.

[0146] The data storage unit 53 is a storage device that uses a combination of non-volatile memory, such as an HDD or memory card, which allows for writing and reading at any time, and volatile memory, such as RAM, as storage media. The data storage unit 53 is used to store data acquired and generated during the process in which the control unit 51 executes a program and performs various processing.

[0147] The communication interface 54 includes one or more wired or wireless communication modules. For example, the communication interface 54 includes a communication module that connects to the central image processing server 10 and the image processing server 20 via a network, either by wire or wirelessly. In other words, the communication interface 54 can be any general communication interface that can communicate with the central image processing server 10 and the image processing server 20 and send and receive various types of information under the control of the control unit 51.

[0148] The input / output interface 55 is connected to the input device 56 and the output device 57, etc. The input / output interface 55 is an interface that enables the transmission and reception of information between the input device 56 and the output device 57. The input / output interface 55 may be integrated with the communication interface 54. For example, the score relay server 50 and at least one of the input device 56 or the output device 57 may be wirelessly connected using short-range wireless technology, and information may be transmitted and received using said short-range wireless technology.

[0149] The input device 56 may include, for example, a keyboard or pointing device for an administrator managing the score relay server 50 to input various information to the score relay server 50. The input device 56 may also include a reader for reading data to be stored in the program storage unit 52 or data storage unit 53 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.

[0150] The output device 57 includes a display or the like that shows various information received from the score relay server 50.

[0151] Next, the software configuration of the score relay server 50 in the embodiment will be described in more detail. The control unit 51 comprises a receiving control unit 511, a list update unit 512, and a transmitting control unit 513.

[0152] The receiving control unit 511 functions as a server list receiving unit that receives list information from the image processing server 20. The receiving control unit 511 stores the received list information in the received information storage unit 531.

[0153] The list update unit 512 updates the list information based on the list information stored in the received information storage unit 531. Details of the method for updating the list information will be described later.

[0154] The transmission control unit 513 functions as a central server list transmission unit that transmits list information. The transmission control unit 513 transmits list information that has been updated by the list update unit 512 or received by the reception control unit 511.

[0155] The data storage unit 53 includes a received information storage unit 531. The received information storage unit 531 is used to store information received by the receiving control unit 511, including list information.

[0156] (operation) Figure 10 is a sequence diagram showing an example of the operation in an image processing system to acquire moving images from a user terminal 30 and display a composite image on the user terminal 30. The sequence shown in Figure 10 is achieved by the control unit 11 of the central image processing server 10, the control unit 21 of the image processing server 20, the control unit 31 of the user terminal 30, the control unit 41 of the image relay router 40, and the score relay server 50 reading and executing programs stored in the program storage units 12, 22, 32, 42, and 52, respectively.

[0157] This sequence is initiated, for example, when the user terminal 30 receives an instruction that the user will participate in an event or multi-point conference, or when the user terminal 30 senses that the start time of the event or multi-point conference has arrived.

[0158] Steps ST201 to ST203 are the same as steps ST101 to ST103, which were explained with reference to Figure 6. Therefore, redundant explanations are omitted here.

[0159] In step ST204, the transmission control unit 213 of the first image processing server 201 transmits list information to the first score relay server 501 via the communication interface 24.

[0160] Steps ST205 to ST207 are the same as steps ST105 to ST107, which were explained with reference to Figure 6. Therefore, redundant explanations are omitted here.

[0161] In step ST208, the transmission control unit 213 of the second image processing server 202 transmits the list information to the first score relay server 501 via the communication interface 24.

[0162] Steps ST209 to ST211 are the same as steps ST109 to ST111, which were explained with reference to Figure 6. Therefore, redundant explanations are omitted here.

[0163] In step ST212, the transmission control unit 213 of the third image processing server 203 transmits the list information to the second score relay server 502 via the communication interface 24.

[0164] In step ST213, the list update unit 512 of the first score relay server 501 updates the list information. The reception control unit 511 of the first score relay server 501 receives list information from the first image processing server 201 and the second image processing server 202. The reception control unit 511 stores the received list information in the reception information storage unit 531. The list update unit 512 updates the list information based on the list information stored in the reception information storage unit 531. The first score relay server 501 receives list information from the first image processing server 201 and the second image processing server 202, and therefore receives lists 1-1 and 1-2. A predetermined number of user IDs with high scores from lists 1-1 and 1-2 are designated as the new list 2-1. An example of list 2-1 is shown below. The predetermined number is set to 4 people in this example.

[0165] List 2-1 User ID, score 1,80 10,90 51, 85 52, 75

[0166] In step ST214, the transmission control unit 513 of the first score relay server 501 transmits list information. The list update unit 512 outputs list information, including list 2-1, to the transmission control unit 513. The transmission control unit 513 transmits the list information to the central image processing server 10 through the communication interface 54.

[0167] In step ST215, the transmission control unit 513 of the second score relay server 502 transmits list information. The reception control unit 511 of the second score relay server 502 receives only the list information from the third image processing server 203. For example, the list received from the third image processing server 203 may be the following list 2-2.

[0168] List 2-2 User ID, score 101, 83 102, 73 111, 53 120, 63

[0169] The receiving control unit 511 outputs list information, including list 2-2, to the transmitting control unit 513. The transmitting control unit 513 transmits the list information to the central image processing server 10 via the communication interface 54.

[0170] In step ST216, the user selection unit 112 selects a predetermined number of users. The user selection unit 112 selects a predetermined number of users from the list 2-1 to list 2-2, starting with the users with the highest scores. The predetermined number can be any number of multiple users; in this case, it is set to 4. From the above list 2-1 to list 2-2, the top 4 users with the highest scores are user IDs 1, 10, 51, and 101. Therefore, the user selection unit 112 generates list 2-3.

[0171] List 2-3 User ID, score 1,80 10,90 51, 85 101, 83

[0172] Thus, similar to the first embodiment, the user selection unit 112 determines that users with user IDs 1, 10, 51, and 101 are the users that should be displayed on the screen. The user selection unit 112 then generates user ID information including user IDs 1, 10, 51, and 101 included in list 1-4 and outputs it to the transmission control unit 113.

[0173] Steps ST217 to ST231 are the same as steps ST114 to ST129, which were explained with reference to Figure 6. Therefore, redundant explanations are omitted here.

[0174] (Effects of the second embodiment) According to the second embodiment, when the score relay server 50 receives list information from multiple image processing servers 20, the score relay server 50 updates the received list information. This makes it possible to reduce the processing load on each server.

[0175] Furthermore, by having the image processing server 20 transmit user image information directly or via the image relay router 40, it becomes possible to send and receive user image information for creating a composite image via the shortest path, thereby reducing the delay when creating the composite image.

[0176] [Modified version of the second embodiment] Figure 11 is a diagram showing an example of an overview of an image processing system according to a modified example of the second embodiment. As shown in Figure 11, the modified version of the second embodiment differs from the second embodiment in that it includes a first central image processing server 101 and a second central image processing server 102 instead of the central image processing server 10, and further includes an algorithm instruction server 60.

[0177] In other words, in the second embodiment, the central image processing server 10 is not limited to one, but may be multiple central image processing servers 10.

[0178] The algorithm instruction server 60 is connected to the first central image processing server 101 and the second central image processing server 102. The algorithm instruction server 60 may be a general-purpose computer. For example, if the algorithm has parameters, the algorithm instruction server 60 may instruct the parameter values ​​of the algorithm to be used by the first central image processing server 101 and the second image processing server 202.

[0179] For example, the central image processing server 10 can change the algorithm by changing parameter values ​​according to instructions from the algorithm instruction server 60. For instance, if the host of a program says, "Now, we will prioritize displaying those from Hokkaido," during video distribution, the algorithm instruction server 60 can send parameter values ​​to the central image processing server 10 that increase the score value of user IDs from Hokkaido, either through voice recognition from the video distribution or through manual operation by the administrator (or operator) of the algorithm instruction server 60.

[0180] The first central image processing server 101 creates a composite image from the video footage of the user terminal 30 managed by the first image processing server 201 and the second image processing server 202. Then, the second central image processing server 102 generates a composite image from the video footage of the user terminal 30 managed by the third image processing server 203.

[0181] For example, in a modified version of the second embodiment, the first central image processing server 101 receives list information including the above-described list 2-1 and determines a user (user ID) for creating a composite image based on list 2-1. Similarly, the second central image processing server 102 determines a user for creating a composite image based on the above-described list 2-2.

[0182] Figure 12 shows an example of a composite image generated according to the instructions of the first central image processing server 101 and the second central image processing server 102. Figure 12(a) is a composite image created by the first image processing server 201 or the second image processing server 202 at the direction of the first central image processing server 101. And Figure 12(b) is a composite image created by the third image processing server 203 at the direction of the second central image processing server 102.

[0183] As shown in Figure 12(a), the first central image processing server 101 transmits user IDs 1, 10, 51, and 52 to the first image processing server 201 and the second image processing server 202 via the image relay router 40, according to List 2-1. Therefore, the first image processing server 201 and the second image processing server 202 generate composite images using the moving images of user IDs 1, 10, 51, and 52, and transmit these composite images to each user (the first user terminal 301 to the 100th user terminal 30100).

[0184] Similarly, as shown in Figure 12(b), the third central image processing server 103 transmits user IDs 101, 102, 111, and 120 to the third image processing server 203 via the image relay router 40, according to List 2-2. The third image processing server 203 then generates a composite image using the moving images of user IDs 101, 102, 111, and 120, and transmits the composite image to each user (user terminal 30101 to user terminal 30150). Returning to Figure 11, in the modified version of the second embodiment, the first image processing server 201 does not need to receive moving images from the third image processing server 203, and similarly, the third image processing server 203 does not need to receive user image information from the first image processing server 201 and the second image processing server 202. In other words, user image information is transmitted only to the second image processing server 202 via the first image relay router 401.

[0185] For example, in a modified version of the second embodiment, steps ST224 and ST226 of the operation shown in Figure 10 can be omitted. That is, the second image relay router 402 becomes unnecessary. Therefore, in a modified version of the second embodiment, the number of relaying image routers can be reduced compared to the first or second embodiment. This makes it possible to reduce the delay when the image processing server 20 creates a composite image.

[0186] [Third Embodiment] (composition) The image processing system of the third embodiment may be the same as that of the second embodiment described with reference to Figure 8, so a redundant explanation will be omitted here.

[0187] However, in the third embodiment, the image processing server 20 calculates multiple scores using multiple algorithms. Therefore, the auxiliary information storage unit 232 may store multiple algorithms. Here, for simplicity, in the third embodiment, we assume that the auxiliary information storage unit 232 stores two algorithms (a first algorithm and a second algorithm).

[0188] The first algorithm may be, for example, an algorithm that calculates a first score using the score calculation method described in the first or second embodiment above. The second algorithm is an algorithm used to calculate the degree of enthusiasm for a particular talent as a second score based on user attribute information. Alternatively, the second algorithm may be one of the predetermined algorithms described above.

[0189] Furthermore, for example, suppose that each user has been surveyed in advance, and the auxiliary information storage unit 232 stores user attribute information, including the responses to the survey. For example, the score calculation unit 212 may use a second algorithm to assign points to each option in the survey responses, and then calculate a score based on the options selected by each user according to the response information.

[0190] Here, user attribute information is information belonging to the user that is obtained from sources other than images and is "not entered by the user during the transmission or reception of composite images, etc." User input information is information belonging to the user that is obtained from sources other than images and is "entered by the user during the transmission or reception of video images, etc."

[0191] Furthermore, user attribute information is not limited to information obtained from questionnaire responses. For example, it may be created based on response information obtained from the user device 30 via another communication means while the transmission control unit 312 of the user device 30 is transmitting the video. In addition, user attribute information may be generated by the control unit 31 of the image processing server 30 based on the characteristics of the video and the communication environment with the user device 30.

[0192] (operation) Figure 13 is a sequence diagram showing an example of the operation in an image processing system to acquire moving images from a user terminal 30 and display a composite image on the user terminal 30. The sequence shown in Figure 13 is realized by the control unit 11 of the central image processing server 10, the control unit 21 of the image processing server 20, the control unit 31 of the user terminal 30, the control unit 41 of the image relay router 40, and the score relay server 50 reading and executing the programs stored in the program storage units 12, 22, 32, 42, and 52, respectively.

[0193] This sequence is initiated, for example, when the user terminal 30 receives an instruction that the user will participate in an event or multi-point conference, or when the user terminal 30 senses that the start time of the event or multi-point conference has arrived.

[0194] Steps ST301 to ST302 are the same as steps ST201 to ST202 explained with reference to Figure 10. Therefore, redundant explanations are omitted here.

[0195] In step ST303, the score calculation unit 212 of the first image processing server 201 uses the first algorithm to calculate a score (first score) representing the degree of matching between the trained image and the acquired video, similar to step ST203. Furthermore, the score calculation unit 212 uses the second algorithm to calculate a score (second score) representing the level of enthusiasm for a specific talent, based on the user attribute information stored in the auxiliary information storage unit 232.

[0196] The score calculation unit 212 of the first image processing server 201 selects a predetermined number of users from the top-performing users for each calculated score. Here, the predetermined number can be any number, but in this case, it will be 4. For example, suppose the user IDs of the above 4 people and the calculated first and second scores are as shown in List 3-1 below. The score calculation unit 212 outputs list information including List 3-1 to the transmission control unit 213.

[0197] List 3-1 User ID, first score, second score 1, 80, 90 2, 70, 80 10, 90, 10 15, 50, 70 20, 40, 60

[0198] As shown in List 3-1, the top four users in the first score are User IDs 1, 2, 10, and 15, and the top four users in the second score are User IDs 1, 2, 15, and 20. Also, as shown in List 3-1, the list information may include multiple scores (the first score and the second score). By including multiple scores at once, the list information only needs to be sent once, thus reducing the amount of data transmitted.

[0199] Steps ST304 to ST306 are the same as steps ST204 to ST206 explained with reference to Figure 10. Therefore, redundant explanations are omitted here.

[0200] In step ST307, the score calculation unit 212 of the second image processing server 202 calculates a first score using the first algorithm, similar to step ST207. Furthermore, the score calculation unit 212 calculates a second score using the second algorithm, similar to step ST303.

[0201] The score calculation unit 212 of the second image processing server 202 selects a predetermined number of users from the top-ranking users for each of the calculated scores, similar to step ST303. For example, suppose the user IDs of the four users and the calculated first and second scores are as shown in List 3-2 below. The score calculation unit 212 outputs list information, including List 3-2, to the transmission control unit 213.

[0202] List 3-2 User ID, first score, second score 51, 85, 85 52, 75, 75 60, 15, 95 65, 55, 65 70, 45, 55

[0203] As shown in List 3-2, the top four users in the first score category are User IDs 51, 52, 65, and 70, and the top four users in the second score category are User IDs 51, 52, 60, and 65.

[0204] Steps ST308 to ST310 are the same as steps ST208 to ST210, which were explained with reference to Figure 10. Therefore, redundant explanations are omitted here.

[0205] In step ST311, the score calculation unit 212 of the third image processing server 203 calculates a first score using the first algorithm, similar to step ST211. Furthermore, the score calculation unit 212 calculates a second score using the second algorithm, similar to step ST303.

[0206] The score calculation unit 212 of the third image processing server 203 selects a predetermined number of users from the top-ranking users for each of the calculated scores, similar to step ST303. For example, suppose the user IDs of the four users and the calculated first and second scores are as shown in List 3-3 below. The score calculation unit 212 outputs list information, including List 3-3, to the transmission control unit 213.

[0207] List 3-3 User ID, first score, second score 101, 83, 93 102, 73, 63 110, 43, 83 111, 53, 73 120, 63, 13

[0208] As shown in List 3-3, the top four users in the first score are User IDs 101, 102, 111, and 120, and the top four users in the second score are User IDs 101, 102, 110, and 111.

[0209] Step ST312 is the same as step ST212, which was explained with reference to Figure 10. Therefore, the redundant explanation here will be omitted.

[0210] In step ST313, the list update unit 512 of the first score relay server 501 updates the list information. The reception control unit 511 of the first score relay server 501 receives the list information from the first image processing server 201 and the second image processing server 202. The reception control unit 511 stores the received list information in the reception information storage unit 531. The list update unit 512 updates the list information based on the list information stored in the reception information storage unit 531. The first score relay server 501 receives lists 3-1 and 3-2 in order to receive list information from the first image processing server 201 and the second image processing server 202. A predetermined number of user IDs with high scores from lists 3-1 and 3-2 are designated as the new list 3-4. An example of list 3-4 is shown below. The predetermined number is set to 4 in this example.

[0211] List 3-4 User ID, first score, second score 1, 80, 90 2, 70, 80 10, 90, 10 51, 85, 85 52, 75, 75 60, 15, 95

[0212] Step ST314 is the same as step ST214, which was explained with reference to Figure 10. Therefore, the redundant explanation here will be omitted.

[0213] In step ST316, the user selection unit 112 selects a predetermined number of users. The user selection unit 112 selects a predetermined number of users from the users with the highest scores among lists 3-3 and 3-4. The predetermined number can be any number of multiple users; in this case, it is set to 4. The combinations of the top 4 users with the highest first and second scores from lists 3-3 and 3-4 are shown in the following list 3-5.

[0214] List 3-5 User ID, first score, second score 1, 80, 90 10, 90, 10 51, 85, 85 60, 15, 95 101, 83, 93

[0215] Referring to List 3-5, the user selection unit 112 determines that the users to be displayed on one screen (users with high first scores) are user IDs 1, 10, 51, and 101, and the users to be displayed on the other screen (users with high second scores) are user IDs 1, 51, 60, and 101. The user selection unit 112 then generates user ID information including the determined user IDs and outputs it to the transmission control unit 113. The user ID information may include information indicating that user IDs 1, 10, 51, 60, and 101 included in List 3-5 will be used for the first composite image, and information indicating that user IDs 1, 51, 60, and 101 will be used for the second composite image.

[0216] Steps ST317 to ST320 are the same as steps ST217 to ST220, which were explained with reference to Figure 10. Therefore, redundant explanations are omitted here.

[0217] In step ST321, the transmission control unit 213 of the first image processing server 201 transmits user image information to the first image relay router 401. Similar to step ST221, the transmission control unit 213 knows that video footage of user ID 1 and user 10 is required based on the information included in List 3-5. The reception control unit 211 of the first image processing server 201 receives user ID information. The transmission control unit 213 then retrieves video footage corresponding to user ID 1 and user ID 10 from the reception information storage unit 231, generates user image information including the user IDs and video footage, and transmits the generated user image information to the first image relay router 401. The user image information includes video footage used in the first composite image and the second composite image. By transmitting the video footage required for the first composite image and the second composite image in a single transmission, the amount of data transmitted can be reduced compared to transmitting multiple times. Furthermore, transmission to the first image relay router 401 may be performed using multicast.

[0218] Steps ST322 to ST323 involve the transmission control units 213 of the second image processing server 202 and the third image processing server generating user image information, including video, corresponding to the user ID information, similar to steps ST222 to ST223, and transmitting the generated user image information to each of the image relay routers 40. Here, in steps ST321 to ST323, similar to steps ST118 to ST120, the transmission control unit 213 may transmit a reduced version of the video.

[0219] Steps ST324 to ST326 are the same as steps ST224 to ST226 explained with reference to Figure 10. Therefore, redundant explanations are omitted here.

[0220] In step ST327, the composite image generation unit 214 of the first image processing server 201 generates a composite image. The receiving control unit 211 of the first image processing server 201 receives user image information and stores the received user image information in the receiving information storage unit 231. The composite image generation unit 214 generates a composite image by combining the video images of the user ID corresponding to List 3-5. The composite image generation unit 214 generates a composite image based on the first score as the first composite image, and generates a composite image based on the second score as the second composite image.

[0221] Figure 14 shows examples of the first and second composite images. As shown in Figure 14(a), the composite image generation unit 214 generates a first composite image by combining video footage corresponding to user ID 1, user ID 10, user ID 51, and user ID 101. Furthermore, as shown in Figure 14(b), the composite image generation unit 214 generates a second composite image by combining video footage corresponding to user ID 1, user ID 51, user ID 60, and user ID 101. These video footage may be combined using video footage from the same time, such as by using timestamps included in the video footage. This ensures that the composite image generated by the composite image generation unit 214 consists of video footage that is time-synchronized.

[0222] In step ST328, the transmission control unit 213 of the first image processing server 201 transmits the composite image (first composite image and second composite image) to the user terminal 30 belonging to the first group user terminal 3a. The composite image generation unit 214 outputs the composite image to the transmission control unit 213. The transmission control unit 213 transmits the first composite image and the second composite image to the first user terminal 301 to the 50th user terminal 3050 (the nth user terminal 30n) belonging to the first group user terminal 3a via the communication interface 24. The reception control unit 313 of each user terminal receives the first composite image and the second composite image and stores the composite image in the reception information storage unit 332. The image control unit 314 then controls the display of the output device 37 to display the first composite image and the second composite image stored in the reception information storage unit 332. As a result, users of user terminal 30 belonging to the first group user terminal 3a can view the first composite image and the second composite image. The first and second composite images may be displayed by splitting the display. Furthermore, if a user instructs the system to display only a specified composite image, only the specified composite image may be displayed.

[0223] (Effects of the third embodiment) According to the third embodiment, by arranging the central image processing server 10 and the image processing server 20, it is possible to reduce the processing load on each server.

[0224] In the third embodiment, where there are multiple algorithms, the user can create and switch between multiple different screens. Furthermore, by generating list information containing multiple scores (e.g., a first score and a second score), it becomes unnecessary to send separate video images to generate multiple composite screens. Therefore, the amount of data transferred can be reduced.

[0225] [Modified example of the third embodiment] In a modified example of the third embodiment, assume that List 3-5 described in the third embodiment was the following List 4-1.

[0226] List 4-1 User ID, first score, second score 1, 80, 90 10, 90, 93 51, 85, 85 60, 15, 95 101, 83, 10

[0227] Focusing on users 1 and 10, the video feeds from both users will be received by the first image processing server 201. Furthermore, both users are among the top four in both the first and second scores. In this case, the video feeds from both user 1 and user 10 will be transmitted by the first image processing server 201, and their transmission paths will be identical. Therefore, by sending user image information, including the video feeds from users 1 and 10, via multicast to the same multicast address, the first image processing server 201 can reduce the redundancy of transmission information in the image relay router 40 on the distribution path and also reduce the processing load.

[0228] For example, the first composite image is a combination of four video clips with user IDs 1, 10, 51, and 101, using four multicast addresses. For example, let's say we use the four addresses 232.0.0.1 to 232.0.0.4. The second composite image is a combination of four video clips with user IDs 1, 10, 51, and 60, using four multicast addresses. For example, let's say we use the four addresses 232.0.0.5 to 232.0.0.8. The contents of the packets from 232.0.0.1 and 232.0.0.5 differ only in their headers; the payload portion is the same, containing the video clip with user ID 1. Similarly, the contents of the packets from 232.0.0.2 and 232.0.0.6 differ only in their headers; the payload portion is the same, containing the video clip with user ID 10. Therefore, the video data from user 1 and user 10 will follow the same path whether they are sending the first or second composite image to the same recipient. Thus, 232.0.0.1 and 232.0.0.2 can be reused, making 232.0.0.5 and 232.0.0.6 redundant. In other words, the amount of data transferred can be reduced to 6 / 8, thus reducing the processing load.

[0229] Furthermore, the video feeds of user 1 and user 10 can be identified by referring to the multicast source address.

[0230] [Differentiation of composite image] Figure 15 shows an example of a composite image used in a modified example of a composite image, and Figure 16 shows an example of a background image used in a modified example of a composite image. In this example, the image processing server 20 may store background information for the background screen in the data storage unit 23. The background information may be received from the central image processing server 10, which is a background image transmission server, or from another server. The control unit 21 of the image processing server 20 generates the screen shown in Figure 15 by combining a foreground image (composite image) which is a composite image obtained by combining the video images of a predetermined number of users with high scores, with the background images stored in the data storage unit 23, and transmits this screen to each of the user terminals 30 belonging to the image processing server 20.

[0231] Furthermore, when the central image processing server 10, which is the background image transmission server, receives a predetermined instruction, the control unit 21 of the image processing server 20 does not combine the foreground image, but instead sends only the background image shown in Figure 16 to each user terminal 30. This makes it possible to make the screen configuration more flexible. For example, it is possible to switch whether or not to display the foreground image that is displayed to multiple users in accordance with the production of a program or other event. Alternatively, the foreground image may be displayed or not displayed in response to instructions from the user. By doing so, user satisfaction is improved.

[0232] [Other embodiments] In the above embodiments, modifications of the second embodiment can also be applied to the first and third embodiments. That is, even in the first embodiment, there may be multiple central image processing servers 10.

[0233] Furthermore, the method described in the above embodiment can be distributed by storing the program (software means) that can be executed by a computer in a storage medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), or semiconductor memory (ROM, RAM, flash memory, etc.), and by transmitting it via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only the execution program but also tables and data structures) to be executed by the computer. The computer realizing this device reads the program stored in the storage medium and, if necessary, constructs the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. The storage medium referred to in this specification is not limited to distribution mediums, but also includes storage mediums such as magnetic disks and semiconductor memory provided inside the computer or in devices connected via a network.

[0234] In short, this invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriately as possible, in which case the combined effects can be obtained. Moreover, the embodiments described above include inventions at various stages, and various inventions can be extracted by appropriate combinations of the multiple constituent elements disclosed. [Explanation of symbols]

[0235] 10…Central Image Processing Server 11…Control Unit 111...Receiver Control Unit 112...User Selection Section 113...Transmission Control Unit 12…Program memory 13…Data storage unit 131...Received information storage unit 132…Selection information storage unit 14…Communication Interface 15… Input / Output Interface 16…Input device 17…Output device 20…Image processing server 21…Control unit 211…Receiving control unit 212…Score calculation unit 213…Transmission control unit 214…Composite image generation unit 22…Program memory unit 23…Data memory unit 231…Received information memory unit 232…Auxiliary information memory unit 24…Communication interface 25…Input / output interface 26…Input device 27…Output device 3…Group user terminal 30…User terminal 31…Control unit 311…Image acquisition unit 312…Transmission control unit 313…Receiving control unit 314…Image control unit 32…Program memory unit 33…Data memory unit 331…Image information memory unit 332…Received information memory unit 34…Communication interface 35…Input / output interface 36…Input device 37…Output device 38…Camera 40…Image relay router 41…Control unit 411…Receiving control unit 412…Transmission control unit 42…Program memory unit 43…Data memory unit 431…Received information memory unit 44…Communication interface ]​​​​​​​​​ 51... Control Unit 511...Receiver Control Unit 512... List update section 513...Transmission Control Unit 52...Program memory unit 53...Data storage unit 531...Received information storage unit 54…Communication Interface 55… Input / Output Interface 56...Input device 57…Output device 60… Algorithm instruction server

Claims

1. Multiple image processing servers, each managing multiple user terminals, The system includes a central image processing server connected to the aforementioned multiple image processing servers, Each of the aforementioned multiple image processing servers is: An image receiving unit that receives image information including moving images captured by the aforementioned multiple user terminals, A memory unit that stores a predetermined algorithm, A score calculation unit calculates a score for each of the received image information using the predetermined algorithm and generates a list containing a predetermined number of user information for the user terminal in descending order of the scores. A list transmission unit that transmits the aforementioned list, Equipped with, The aforementioned central image processing server is A list receiving unit that receives the list from the aforementioned multiple image processing servers, A user selection unit that selects user information to be used to create images to be displayed on each of the multiple user terminals from the list received from the multiple image processing servers, A user information transmission unit that transmits the selected user information to each of the multiple image processing servers, An image processing system equipped with the following features.

2. The aforementioned list includes scores corresponding to user information, The image processing system according to claim 1, wherein the user selection unit selects a predetermined number of user information from the list in descending order of scores.

3. The aforementioned image processing server is A user information receiving unit that receives the user information, A video transmission unit that transmits video images from a user terminal specified by the user information to another image processing server, A video receiving unit that receives video from the aforementioned other image processing server of a user terminal managed by the other image processing server specified by the user information, A composite image generation unit that generates a composite image by combining video images received from the aforementioned other image processing server and video images transmitted to the other image processing server, A user terminal transmission unit that transmits the composite image to multiple user terminals managed by the image processing server, The image processing system according to claim 1, further comprising:

4. The image processing system according to claim 3, wherein the video transmission unit reduces the size of the video to be transmitted to the other image processing server and transmits it.

5. The aforementioned image processing system is The system further includes an image relay router, and the image relay router is: A user information relay unit that receives the user information from the central image processing server and transmits it to at least one of the plurality of image processing servers, A video relay unit that receives video from the image processing server that sent the user information and transmits the received video to another video relay router, The image processing system according to claim 1, further comprising:

6. The aforementioned image processing system is The system further comprises one or more image processing servers and score relay servers connected to the central image processing server, The score relay server connected to the aforementioned multiple image processing servers is: A server list receiving unit that receives the list from each of the aforementioned multiple image processing servers, A list update unit selects user information from the aforementioned list to be used to create images to be displayed on each of the multiple user terminals, and generates an update list containing the selected user information. A central server list transmission unit that transmits the update list to the central image processing server, The image processing system according to claim 1, comprising:

7. The memory unit further stores the learned video images, The image processing system according to claim 1, wherein the score calculation unit calculates the degree of matching between the video image and the learned video image as the score.

8. The aforementioned predetermined algorithm comprises at least two algorithms, The memory unit further stores learned video images and user attribute information. The score calculation unit calculates the degree of matching between the video and the trained video as a first score using one of the at least two algorithms. A second score is calculated based on the user attribute information using one of the other two algorithms described above. The image processing system according to claim 1, which generates a list containing a predetermined number of pairs of user information of the user terminal and the first score, in descending order of the first score, and a predetermined number of pairs of user information of the user terminal and the second score, in descending order of the second score.

9. An image processing method executed by the processors of multiple image processing servers that manage multiple user terminals and the processor of a central image processing server connected to the multiple image servers, Each of the processors in the aforementioned plurality of image processing servers is: Receiving image information including moving images captured by the aforementioned multiple user terminals, To store a predetermined algorithm, The process involves calculating a score for each of the received image pieces using the predetermined algorithm, To generate a list containing a predetermined number of user information for the user terminals, sorted by the highest score, Sending the aforementioned list, The central image processing server's processor is equipped with, Receiving the list from the aforementioned multiple image processing servers, Select user information from the list received from the multiple image processing servers to be used to create images to be displayed on each of the multiple user terminals, The selected user information is transmitted to each of the multiple image processing servers, An image processing method comprising:

10. An image processing program comprising instructions to be executed by the processors of multiple image processing servers that manage multiple user terminals and the processor of a central image processing server connected to the multiple image processing servers, wherein the instructions to be executed by each of the processors of the multiple image processing servers are: Receiving image information including moving images captured by the aforementioned multiple user terminals, To store a predetermined algorithm, The process involves calculating a score for each of the received image pieces using the predetermined algorithm, To generate a list containing a predetermined number of user information for the user terminals, sorted by the highest score, Sending the aforementioned list, The instructions to be executed by the processor of the central image processing server are: Receiving the list from the aforementioned multiple image processing servers, Select user information from the list received from the multiple image processing servers to be used to create images to be displayed on each of the multiple user terminals, The selected user information is transmitted to each of the multiple image processing servers, An image processing program equipped with the following features.

Images