SERVER DEVICE, SYSTEM, AND SYSTEM OPERATION METHOD

Abstract

To provide a sever device, a system, and a system operating method, which provide smoother communication among users participating in a virtual event and an improved user experience.SOLUTION: In a virtual event provision system 1 disclosed herein, a server device 10 comprises a communication unit 101 and a control unit 103 configured to communicate using the communication unit 101. The control unit 103 receives mode information indicative of a participation mode of each of multiple users in a virtual event from terminal devices 12 of the users, and sends information for generating a virtual event image having an image of each user placed at a position with a priority level corresponding to a participation mode to the terminal devices 12.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a server device, a system, and a method of operating the system. [Background technology]

[0002] There is a known method in which computers at multiple locations communicate via a network and virtual events such as conferences are held in a virtual space on the network. Various techniques have been proposed to support smooth communication between users in such virtual events. For example, Patent Document 1 discloses a system that corrects the image of the other party displayed on the computer used by each user to look at the camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6849133 Summary of the Invention [Problem to be solved by the invention]

[0004] There is room to facilitate smoother communication between users participating in virtual events on the network and improve the user experience.

[0005] The present disclosure provides a server device etc. that contributes to improving the user experience of users participating in a virtual event. [Means for solving the problem]

[0006] The server device in the present disclosure has a communication unit and a control unit that communicates via the communication unit, and the control unit receives behavior information indicating the participation behavior of each of a plurality of users in a virtual event from each user's terminal device, and sends information to the terminal device for generating an image of the virtual event in which an image of each user is positioned at a priority position according to each user's participation behavior based on the behavior information.

[0007] The system of the present disclosure is a system having a server device and a terminal device that communicate with each other, wherein the terminal device sends behavior information indicating the participation behavior of each of a plurality of users in a virtual event to the server device, and the server device sends information to the terminal device for generating an image of the virtual event in which an image of each user is positioned at a priority position according to each user's participation behavior based on the behavior information.

[0008] A system operation method in the present disclosure is a system operation method having a server device and a terminal device that communicate with each other, and includes a step in which the terminal device sends behavior information indicating the participation behavior of each of a plurality of users in a virtual event to the server device, and a step in which the server device sends information to the terminal device based on the behavior information for generating an image of the virtual event in which an image of each user is positioned at a priority position according to each user's participation behavior. Effect of the Invention

[0009] According to the terminal device etc. of the present disclosure, it is possible to contribute to improving the user experience of users participating in a virtual event. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a virtual event providing system. [Diagram 2] FIG. 11 is a sequence diagram showing an example of the operation of the virtual event providing system. [Figure 3A] FIG. 11 is a flowchart illustrating an example of the operation of the terminal device. [Figure 3B] FIG. 11 is a flowchart illustrating an example of the operation of the server device. [Figure 3C] FIG. 11 is a flowchart illustrating an example of the operation of the terminal device. [Figure 4A] FIG. 13 is a diagram illustrating an example of a virtual event image. [Figure 4B] FIG. 13 is a diagram illustrating an example of a virtual event image. [Figure 4C] FIG. 13 is a diagram illustrating an example of a virtual event image. [Figure 4D] FIG. 13 is a diagram illustrating an example of a virtual event image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The following describes the embodiments.

[0012] FIG. 1 is a diagram showing an example of the configuration of a virtual event providing system in one embodiment. The virtual event providing system 1 has a server device 10 and a plurality of terminal devices 12 connected to each other so as to be able to communicate information with each other via a network 11. The virtual event providing system 1 is a system for providing an event in a virtual space in which a user can participate using the terminal device 12, that is, a virtual event. A virtual event is an event in which a plurality of participants communicate information by speech or the like in a virtual space, and each participant is represented by a user image such as a 2D image or a 3D model that represents each participant. The virtual event in this embodiment is a discussion between participants on an arbitrary theme.

[0013] The server device 10 is, for example, a server computer that belongs to a cloud computing system or other computing system and functions as a server that implements various functions. The server device 10 may be composed of two or more server computers that are connected to each other so as to be able to communicate information and operate in cooperation with each other. The server device 10 transmits and receives information and processes information required to provide a virtual event.

[0014] The terminal device 12 is an information processing device equipped with a communication function, and is used by a user (hereinafter, referred to as a participant) who participates in a virtual event provided by the server device 10. The terminal device 12 is, for example, an information processing device such as an information processing terminal, such as a smartphone or a tablet terminal, or a personal computer.

[0015] Network 11 may be, for example, the Internet, but may also include an ad-hoc network, a Local Area Network (LAN), a Metropolitan Area Network (MAN), or other network, or any combination thereof.

[0016] In this embodiment, the server device 10 has a communication unit 101 and a control unit 103 that communicates through the communication unit 101. The control unit 103 receives behavior information indicating the participation behavior of each of the multiple users in the virtual event from the terminal device 12 of each user, and transmits information for generating an image of the virtual event (hereinafter referred to as a virtual event image) in which the image of each user (hereinafter referred to as a user image) is arranged at a position of priority according to each participation behavior based on the behavior information to the terminal device 12. The participation behavior includes attention of each user to the user image of other users in the virtual event image, and the control unit 103 determines the priority according to the amount of attention from other users. That is, the user image of a user who has a larger amount of attention from other users is arranged at a position of higher priority. Alternatively, the participation behavior includes speech of each user in the virtual event, and the control unit 103 determines the priority according to the amount of speech of each user. That is, the user image of a user who has a larger amount of speech is arranged at a position of higher priority. The terminal device 12 displays the virtual event image configured in this way to the user. Each user can communicate by viewing the virtual event image of a user who is highly popular or who speaks a lot, which is placed in a high priority position, for example, in the center of the image. This allows communication to be intuitively focused on the user who is dominant in attention or conversation, which makes communication smoother and improves the user experience.

[0017] The configurations of the server device 10 and the terminal device 12 will now be described in detail.

[0018] The server device 10 includes a communication unit 101, a storage unit 102, a control unit 103, an input unit 105, and an output unit 106. When the server device 10 is configured with two or more server computers, these components are appropriately arranged in the two or more computers.

[0019] The communication unit 101 includes one or more communication interfaces. The communication interface is, for example, a LAN interface. The communication unit 101 receives information used in the operation of the server device 10, and transmits information obtained by the operation of the server device 10. The server device 10 is connected to a network 11 by the communication unit 101, and communicates information with a terminal device 12 via the network 11.

[0020] The storage unit 102 includes, for example, one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of them, functioning as a main storage device, an auxiliary storage device, or a cache memory. The semiconductor memory is, for example, a random access memory (RAM) or a read only memory (ROM). The RAM is, for example, a static RAM (SRAM) or a dynamic RAM (DRAM). The ROM is, for example, an electrically erasable programmable ROM (EEPROM). The storage unit 102 stores information used in the operation of the server device 10 and information obtained by the operation of the server device 10.

[0021] The control unit 103 includes one or more processors, one or more dedicated circuits, or a combination of these. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit), or a dedicated processor such as a GPU (Graphics Processing Unit) specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc. The control unit 103 executes information processing related to the operation of the server device 10 while controlling each unit of the server device 10.

[0022] The input unit 105 includes one or more input interfaces. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen integrated with a display, or a microphone that accepts voice input. The input unit 105 accepts an operation to input information used in the operation of the server device 10, and sends the input information to the control unit 103.

[0023] The output unit 106 includes one or more output interfaces. The output interface is, for example, a display or a speaker. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. The output unit 106 outputs information obtained by the operation of the server device 10.

[0024] The functions of the server device 10 are realized by a processor included in the control unit 103 executing a control program. The control program is a program for causing a computer to function as the server device 10. Also, some or all of the functions of the server device 10 may be realized by a dedicated circuit included in the control unit 103. Also, the control program may be stored in a non-transitory recording / storage medium readable by the server device 10, and read by the server device 10 from the medium.

[0025] The terminal device 12 includes a communication unit 111 , a storage unit 112 , a control unit 113 , an input unit 115 , an output unit 116 , and an imaging unit 117 .

[0026] The communication unit 111 has a communication module compatible with wired or wireless LAN standards, a module compatible with mobile communication standards such as LTE, 4G, 5G, etc. The terminal device 12 is connected to the network 11 by the communication unit 111 via a nearby router device or a mobile communication base station, and performs information communication with the server device 10, etc. via the network 11.

[0027] The memory unit 112 includes one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these. The semiconductor memories are, for example, RAM or ROM. The RAM is, for example, SRAM or DRAM. The ROM is, for example, EEPROM. The memory unit 112 functions as, for example, a main memory device, an auxiliary memory device, or a cache memory. The memory unit 112 stores information used in the operation of the control unit 113 and information obtained by the operation of the control unit 113.

[0028] The control unit 113 has, for example, one or more general-purpose processors such as a CPU, an MPU (Micro Processing Unit), or one or more dedicated processors such as a GPU specialized for a specific process. Alternatively, the control unit 113 may have one or more dedicated circuits such as an FPGA, an ASIC, or the like. The control unit 113 generally controls the operation of the terminal device 12 by operating according to a control / processing program, or operating according to an operating procedure implemented as a circuit. The control unit 113 transmits and receives various information to and from the server device 10, etc. via the communication unit 111, and executes the operation according to this embodiment.

[0029] The input unit 115 includes one or more input interfaces. The input interfaces include, for example, physical keys, capacitive keys, a pointing device, and a touch screen that is integral with a display. The input interfaces also include a microphone that accepts voice input, and a camera that captures captured images. The input interfaces may also include a scanner or camera that scans image codes, and an IC card reader. The input unit 115 accepts an operation to input information used in the operation of the control unit 113, and sends the input information to the control unit 113.

[0030] The output unit 116 includes one or more output interfaces. The output interfaces include, for example, a display and a speaker. The display is, for example, an LCD or an organic EL display. The output unit 116 outputs information obtained by the operation of the control unit 113.

[0031] The imaging unit 117 includes a camera that captures an image of a subject using visible light, and a distance measurement sensor that measures the distance to the subject to obtain a distance image. The camera captures images of the subject at, for example, 15 to 30 frames per second to generate a video consisting of continuous captured images. The distance measurement sensor includes a ToF (Time Of Flight) camera, LiDAR (Light Detection And Ranging), and a stereo camera, and generates a distance image of the subject including distance information. The imaging unit 117 sends the captured image and the distance image to the control unit 113.

[0032] The functions of the control unit 113 are realized by a processor included in the control unit 113 executing a control program. The control program is a program for causing the processor to function as the control unit 113. Also, some or all of the functions of the control unit 113 may be realized by a dedicated circuit included in the control unit 113. Also, the control program may be stored in a non-transitory recording / storage medium readable by the terminal device 12, and read by the terminal device 12 from the medium.

[0033] In this embodiment, the control unit 113 acquires a captured image and a distance image of the user of the terminal device 12 by the imaging unit 117, and collects the user's speech by the microphone of the input unit 115. The control unit 113 generates encoded information by encoding the captured image and the distance image of the user for generating a user image, and audio information for reproducing the user's voice. The control unit 113 may perform any processing (e.g., resolution change and trimming) on ​​the captured image, etc., during encoding. The control unit 113 transmits the encoded information to the other terminal device 12 via the server device 10 by the communication unit 111. The control unit 113 also receives the encoded information transmitted from the other terminal device 12 via the server device 10 by the communication unit 111. When the control unit 113 decodes the encoded information received from the other terminal device 12, it uses the decoded information to generate a user image representing a user using the other terminal device 12, and places the user image in the virtual space together with the user image of the user. When the control unit 113 generates a virtual space image for output including a user image from a predetermined viewpoint in the virtual space, that is, a virtual event image, by rendering, the output unit 116 displays the virtual event image and outputs speech based on the voice information of each user. By such operations of the control unit 113 etc., the user of the terminal device 12 can participate in a virtual event in real time and converse with other users.

[0034] 2 is a sequence diagram for explaining the operation procedure of the virtual event providing system 1. This sequence diagram shows the procedure related to the cooperative operation of the server device 10 and multiple terminal devices 12 (terminal devices 12A and 12B are used to distinguish between them). The terminal device 12A is used by a user who is the administrator of the virtual event. The multiple terminal devices 12B are used by users other than the administrator. The operation procedure related to the terminal device 12B shown here is executed by each terminal device 12B, or by each terminal device 12B and the server device 10.

[0035] 2 are executed by the respective control units 103 and 113. Moreover, the steps relating to the transmission and reception of various information in the server device 10 and the terminal device 12 are executed by the respective control units 103 and 113 transmitting and receiving information to each other via the communication units 101 and 111, respectively. In the server device 10 and the terminal device 12, the control units 103 and 113 store the information to be transmitted and received in the storage units 102 and 112, respectively, as appropriate. Moreover, the control unit 113 of the terminal device 12 accepts input of various information through the input unit 115 and outputs various information through the output unit 116.

[0036] In step S200, the terminal device 12A accepts input of setting information for the virtual event by the administrator user. The setting information includes the schedule of the virtual event, the discussion topic, a list of participants, and the like. The list of participants includes the names of participants and email addresses of each participant. Then, in step S201, the terminal device 12A sends the setting information to the server device 10. The server device 10 receives the information sent from the terminal device 12A. For example, the terminal device 12A accesses a site for implementing the virtual event provided by the server device 10, acquires an input screen for setting information, and displays the input screen. Then, the administrator user inputs the setting information into the input screen, and the setting information is sent to the server device 10.

[0037] In step S202, the server device 10 sets up a virtual event based on the setting information. The control unit 103 stores the information on the virtual event and the information on the prospective participants in the storage unit 102 in association with each other.

[0038] In step S203, the server device 10 sends authentication information to each of the terminal devices 12B. The authentication information is information for identifying and authenticating a user who uses the terminal device 12B, and is information such as an ID and a passcode used when participating in a virtual event. For example, this information is sent as an attachment to an e-mail. The terminal devices 12B receive the information sent from the server device 10.

[0039] In step S205, the terminal device 12B sends the authentication information and participation application information received from the server device 10 to the server device 10. The user of the terminal device 12B operates the terminal device 12B to apply for participation in the virtual event using the authentication information sent from the server device 10. For example, the terminal device 12B accesses a site for the virtual event provided by the server device 10, obtains an input screen for the authentication information and information for the participation application, and displays the input screen to the user. Then, the terminal device 12B accepts the information input by the user and sends it to the server device 10.

[0040] In step S206, the server device 10 authenticates the user and completes the participation acceptance. The storage unit 102 stores the identification information of the terminal device 12B and the identification information of the user in association with each other.

[0041] In steps S208 and S209, the server device 10 sends a virtual event start notification to the terminal devices 12A and 12B, respectively. Upon receiving the information sent from the server device 10, the terminal devices 12A and 12B each start capturing an image of the user and collecting the spoken voice.

[0042] In step S210, a virtual event is held by the terminal devices 12A and 12B via the server device 10. Each terminal device 12 transmits and receives information for generating each user image and information on speech sounds to each other via the server device 10. In addition, each terminal device 12 outputs a virtual event image including the user's own and other user images and the speech sounds of other users to the user.

[0043] 3A to 3C, the operation procedures of the server device 10 and the terminal device 12 involved in implementing a virtual event will be described. Figures 3A and 3C are flow charts showing an example of the operation procedures of the terminal device 12. Also, Figure 3B is a flow chart showing an example of the operation procedures of the server device 10.

[0044] FIG. 3A relates to an operation procedure of the control unit 113 when each terminal device 12 transmits information for generating a user image of the user using that terminal device 12.

[0045] In step S302, the control unit 113 controls the imaging unit 117 to capture a visible light image and a distance image of the participant at an arbitrarily set frame rate, and collects the speech of the participant via the input unit 115. The control unit 113 acquires the visible light captured image and the distance image from the imaging unit 117, and acquires audio information from the input unit 115.

[0046] In step S303, the control unit 113 generates aspect information using the captured image, the distance image, and the audio information.

[0047] The aspect information is, for example, information for identifying a user image of another user that the user pays attention to. By executing the procedure of FIG. 3C described later, the terminal device 12 displays a virtual event image toward the user. The virtual event image includes user images showing the user and the user of the other terminal device 12. The control unit 113 identifies a user image that the user pays attention to in the virtual event image. For example, the control unit 113 performs image processing using a captured image and a distance image of the user, and detects the user's gaze point in the virtual event image. The control unit 113 detects the user's gaze point using information such as the position of the user image in the virtual event image, the position of the display and the camera on which the virtual event image is displayed, and the distance from the display and the camera to the position of the user's eyes, and identifies the user image of the other user that corresponds to the gaze point.

[0048] Moreover, the manner information is, for example, information on the amount of speech of the user. The amount of speech is, for example, the total amount of speech during the most recent determination period (for example, several seconds to several minutes). The control unit 113 detects, as speech, a sound having a sound pressure equal to or higher than an arbitrary reference sound pressure in a frequency band (for example, 100 Hz to 1000 Hz) to which human speech belongs. The control unit 113 may distinguish a sound matching a preset language from other noises by voice recognition. The control unit 113 derives the amount of speech by accumulating the time during which speech is detected during the determination period.

[0049] In step S304, the control unit 113 encodes the captured image, the distance image, the audio information, and the aspect information to generate encoded information.

[0050] In step S306, the control unit 113 packetizes the encoded information using the communication unit 111, and transmits the packetized information to the server device 10 and other terminal devices 12.

[0051] When the control unit 113 acquires information input corresponding to a user operation for interrupting imaging and sound collection or for exiting the virtual event (Yes in S308), it terminates the processing procedure of FIG. 3A. If the control unit 113 does not acquire information corresponding to an operation for interrupting or exiting (No in S308), it executes steps S302 to S306 and transmits information for generating a user image and information for outputting sound, together with mode information, to the server device 10 for other terminal devices 12.

[0052] Fig. 3B relates to an operation procedure of the control unit 103 when the server device 10 relays information sent by the terminal device 12. When the control unit 103 receives a packet sent by the terminal device 12 by executing the procedure of Fig. 3A, it executes steps S310 to S318.

[0053] In step S310, the control unit 103 decodes the encoded information contained in the packet received from the terminal device 12 to obtain a captured image, a distance image, audio information, and aspect information.

[0054] In step S312, the control unit 103 determines the priority of the user images based on the aspect information. For example, the control unit 103 derives other user images that each user pays attention to for each of the multiple terminal devices 12. Then, the control unit 103 counts up the number of other users that each user pays attention to for each user image, and determines the priority in descending order of the counting result, that is, in descending order of the amount of attention. Also, for example, the control unit 103 determines the priority of the user images representing the users of the multiple terminal devices 12 in descending order of the amount of speech. In this way, the control unit 103 determines the priority of the user images of users participating in the virtual event according to the amount of attention or the amount of speech. In other words, the user images of users who are more dominant in the attention they receive from other users or in conversations with other users in the virtual event are assigned a higher priority.

[0055] In step S314, the control unit 103 determines the arrangement of each user image in the event image according to the priority of each user image. The arrangement according to the priority is determined based on a rule arbitrarily set in advance. For example, the control unit 103 determines the arrangement of the user images such that the higher the priority of the user image, the closer the user image is to the center of the virtual event image. The control unit 103 may also determine the arrangement of the user images such that the higher the priority of the user image, the closer the user image is to the top of the virtual event image. In this case, the user images are arranged so as to form a hierarchy according to the priority, for example.

[0056] In step S316, the control unit 103 encodes the captured image, the distance image, the audio information, and the layout information of the user image to generate encoded information.

[0057] In step S 318 , the control unit 113 causes the communication unit 101 to packetize the encoded information and transmit it to the other terminal device 12 .

[0058] Fig. 3C relates to an operation procedure of the control unit 113 when the terminal device 12 outputs an image of a virtual event and the voice of another user. When the control unit 113 receives a packet transmitted by another terminal device 12 executing the procedure of Fig. 3A via the server device 10 executing the procedure of Fig. 3B, it executes steps S320 to S323.

[0059] In step S320, the control unit 113 obtains a captured image, a distance image, audio information, and position information by decoding the encoded information included in the packet received from the other terminal device 12. In addition, when executing step S302, the control unit 113 obtains a captured image and a distance image of its own user from the imaging unit 117 and audio information from the input unit 115.

[0060] In step S322, the control unit 113 generates user images of the user and other users based on the captured image and the distance image. The user images are, for example, 2D images of the face, upper body, etc. of each user, 3D models, character images obtained by converting the captured image by an arbitrary algorithm, etc.

[0061] When receiving information from terminal devices 12 of a plurality of users, the control unit 113 executes steps S320 to S322 for each of the terminal devices 12, and generates a user image for each user.

[0062] In step S323, the control unit 113 arranges each user image in the virtual space where the virtual event is held. Coordinate information of the virtual space and information on coordinates where each user image should be initially arranged according to, for example, the order of authentication are stored in the storage unit 112 in advance. In addition, when the control unit 113 acquires position information generated in the server device 10, it arranges each user image based on the arrangement information.

[0063] In step S324, the control unit 113 generates, by rendering, a virtual space image obtained by capturing a plurality of user images arranged in a virtual space from a virtual viewpoint.

[0064] In step S326, control unit 113 displays a virtual space image, i.e., a virtual event image, and outputs sound by output unit 116. That is, control unit 113 outputs information for displaying the virtual event image to output unit 116, and output unit 116 displays the virtual event image and outputs sound.

[0065] By repeatedly executing steps S320 to S326 by the control unit 113, the user can listen to the voices of the other users while watching the moving images of the virtual event images including the user's and other users' images. At this time, the images of the users are displayed in an arrangement according to the participation mode.

[0066] 4A to 4D show examples of virtual event images displayed on the terminal device 12. FIG.

[0067] FIG. 4A is an example of a virtual event image 400 in which user images 40 to 46 are initially arranged.

[0068] FIG. 4B is an example of a virtual event image 400 in which user images 40 to 46 are arranged based on the arrangement information. Here, the user image 40 of the most dominant user in terms of attention acquisition or conversation is arranged in the central area with the highest priority, that is, inside the boundary 48. The user images 41 and 42 of the second most dominant user are arranged on the periphery of the central area, that is, between the boundaries 48 and 49. The user images 43, 44, 45, and 46 of the least dominant user are arranged outside the boundary 49. By arranging them in this way, a user who views the virtual event image 400 can intuitively focus on the user image of the dominant user in terms of attention acquisition or conversation, which allows for smoother communication to be improved.

[0069] FIG. 4C is an example of a virtual event image 400 in which the dominant user in attention acquisition or conversation changes as the participation manner of each user changes. FIG. 4C shows an example in which the user of user image 42 becomes more dominant than the user of user image 40, which was the most dominant in FIG. 4B, and the user images 40 and 42 are replaced accordingly. Here, a case is shown in which the user image 40 moves from the inside to the outside of the boundary 48 of the central area (arrow 40B), and the user image 42 moves from the outside to the inside of the boundary 48 (arrow 40A). In this way, the arrangement of the user images changes dynamically in response to the change in the participation manner of each user, so that a user viewing the virtual event image 400 can intuitively grasp the change in the dominant user in attention acquisition or conversation.

[0070] FIG. 4D is an example of a virtual event image 400 in which the user images 40 to 46 are arranged in another manner based on the arrangement information. Here, the user image 40 of the most dominant user in attention or conversation is arranged in the highest layer with the highest priority, that is, above the boundary 48. The user images 41 and 42 of the next most dominant user are arranged in the middle layer, that is, between the boundaries 48 and 49. The user images 43, 44, 45, and 46 of the least dominant user are arranged in the lowest layer, that is, below the boundary 49. Even with such an arrangement, a user who views the virtual event image 400 can intuitively concentrate on the user image of the user who is dominant in attention acquisition or conversation, thereby enabling smooth improvement of communication.

[0071] As a modified example, instead of the terminal device 12 executing step S303 in FIG. 3A, the server device 10 in FIG. 3B may generate behavior information based on captured image or audio information for each terminal device 12 after step S310.

[0072] Furthermore, this embodiment also includes a case where the priority order for determining the arrangement of user images is determined based on the amount of attention and speech from other users. For example, the server device 10 or the terminal device 12 can normalize the amount of attention and the amount of speech to an arbitrary score and determine the priority order in the order of the total score. Alternatively, the total may be calculated by applying an arbitrary weight to each of the scores of the amount of attention and the amount of speech.

[0073] In the above description, an example has been given in which the layout of the user image is divided into three levels, but the number of levels is not limited to this.

[0074] In the above, the embodiment has been described based on the drawings and examples, but it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these modifications and corrections are included in the scope of the present disclosure. For example, the functions included in each means, each step, etc. can be rearranged so as not to be logically inconsistent, and multiple means, steps, etc. can be combined into one or divided into one. [Explanation of symbols]

[0075] 1. Virtual Event Delivery System 10. Server device 11 Network 12 Terminal Equipment 101, 111 Communications Department 102, 112 Storage section 103, 113 Control section 105, 115 Input section 106, 116 Output section 117 Imaging unit

Claims

1. The Communications Department and A control unit that performs communication by the communication unit, the control unit receives behavior information indicating a participation behavior of each of a plurality of users in a virtual event from a terminal device of each user, and sends information to the terminal device, based on the behavior information, for generating an image of the virtual event in which an image of each user is arranged at a position of priority according to each participation behavior of the user; The aspect information includes information that identifies an image of another user corresponding to the gaze point of the user in the image of the virtual event displayed on the terminal device, the gaze point of the user being identified by image processing using a captured image of the user of the terminal device, The control unit is tallying up the number of other users who pay attention to each image of the user based on the behavior information as an attention amount; A higher priority is set for each user as the amount of attention increases. Server device.

2. In claim 1, The participation manner includes utterances of each user in the virtual event; The control unit determines the priority according to an amount of speech of each user. Server device.

3. In claim 1, The control unit changes a priority of an image of each user according to the aspect information during the virtual event, and sends information for generating an image of the virtual event corresponding to the changed priority to the terminal device. Server device.

4. A system having a server device and a terminal device that communicate with each other, the terminal device transmits to the server device behavior information indicating a behavior of each of the plurality of users in the virtual event; the server device transmits to the terminal device, based on the aspect information, information for generating an image of the virtual event in which an image of each user is arranged at a position of priority according to each participation aspect of the user; The aspect information includes information that identifies an image of another user corresponding to the gaze point of the user in the image of the virtual event displayed on the terminal device, the gaze point of the user being identified by image processing using a captured image of the user of the terminal device, The server device includes: tallying up the number of other users who pay attention to each image of the user based on the behavior information as an attention amount; A higher priority is set for each user as the amount of attention increases. system.

5. In claim 4, The participation manner includes an utterance of each user in the virtual event, and the priority is determined according to an amount of utterance of each user. system.

6. In claim 4, The server device or the terminal device changes a priority of an image of each user according to the aspect information during the virtual event; the terminal device outputs an image of the virtual event based on information for generating an image of the virtual event corresponding to the changed priority. system.

7. 1. A method of operating a system having a server device and a terminal device in communication with each other, comprising: a step of transmitting, by the terminal device, behavior information indicating a participation behavior of each of a plurality of users in a virtual event to the server device; The server device transmits to the terminal device information for generating an image of the virtual event in which an image of each user is arranged at a position of priority according to the participation mode of each user, based on the mode information; The aspect information includes information that identifies an image of another user corresponding to the gaze point of the user in the image of the virtual event displayed on the terminal device, the gaze point of the user being identified by image processing using a captured image of the user of the terminal device, The method of operation includes: The server device, a step of counting the number of other users who pay attention to each image of the user based on the behavior information as an attention amount; and setting a higher priority for each of the users as the amount of attention increases. How it works.

8. In claim 7, The participation manner includes an utterance of each user in the virtual event, and the priority is determined according to an amount of utterance of each user. How it works.

9. In claim 7, The server device or the terminal device changes a priority of an image of each user according to the aspect information during the virtual event; The terminal device further includes a step of outputting an image of the virtual event based on information for generating an image of the virtual event corresponding to the changed priority. How it works.

Images