Computer program, method, system, and server device

Abstract

To provide a scheme for displaying moving-images using a tracking device, for example, mounted to the body of a streamer.SOLUTION: A computer program, by being executed by at least one processor mounted to a terminal device, causes the at least one processor to: operate on the basis of motion of a user of the terminal device him or herself; display a first moving-image related to the real space visible from the viewpoint of the user him or herself; and display a second moving-image related to the animation of an avatar object based on motion of a first user different from the user him or herself, in combination with the first moving-image, when it is determined that the user him or herself has executed a first prescribed action in the first moving-image.SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] The technology disclosed in this application relates to a computer program, a method, a system, a server device, and the like. [Background technology]

[0002] BACKGROUND ART Conventionally, there has been known an information processing system that distributes moving images including animation of an avatar object based on the movement of a user (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-184689 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, various tracking devices, such as head-mounted displays (HMDs) and smart glasses, that are worn on the bodies of broadcasters are likely to be used as terminal devices used by viewers.

[0005] Therefore, the technology disclosed in this application provides a method for displaying moving images using a tracking device that is worn on the viewer's body. [Means for solving the problem]

[0006] A computer program according to one embodiment can "be executed by at least one processor installed in a terminal device, to cause the at least one processor to function in such a way that it operates based on the actions of the user who is the user of the terminal device, displays a first video of the real space as seen from the viewpoint of the user, and, when it is determined that the user has performed a first predetermined action in the first video, displays a second video of an animation of an avatar object based on the actions of a first user different from the user in combination with the first video."

[0007] A method according to one embodiment can be described as "a method executed by at least one processor mounted on a terminal device that executes computer-readable instructions, wherein the at least one processor executes the instructions to operate based on the actions of a user who is the user of the terminal device, and to display a first video of the real space as seen from the viewpoint of the user, and when it is determined that the user has performed a predetermined action in the first video, to display a second video of an animation of an avatar object based on the actions of a first user different from the user, in combination with the first video."

[0008] A system according to one embodiment may be "a system including a first terminal device and a server device connected to the first terminal device via a communication line, wherein the server device is configured to transmit to the first terminal device first movement data generated based on the movement of a second user different from the first user of the first terminal device, received from a second terminal device of the second user; the first terminal device is configured to receive the first movement data from the server device, acquire second movement data generated based on the movement of the first user, operate based on the second movement data, display a first video of real space as seen from the viewpoint of the first user, and, when it is determined that the first user has performed a predetermined action in the first video, display a second video of animation of an avatar object based on the first movement data in combination with the first video."

[0009] A method according to another aspect can be "a method executed by a system including a first terminal device and a server device connected to the first terminal device via a communication line, the method including: the server device transmitting to the first terminal device first movement data generated based on the movement of a second user different from the first user of the first terminal device, the first movement data being received from a second terminal device of the second user; the first terminal device receiving the first movement data from the server device; the first terminal device acquiring second movement data generated based on the movement of the first user; the first terminal device operating based on the second movement data and displaying a first video of real space as seen from the viewpoint of the first user; and, when it is determined that the first user has performed a predetermined action in the first video, the first terminal device displaying a second video of animation of an avatar object based on the first movement data in combination with the first video."

[0010] A server device according to one embodiment can be configured as follows: "A server device having at least one processor, wherein the at least one processor is configured to send a web page including a computer program to a terminal device, wherein the computer program, when executed by the terminal device that receives the web page, operates based on the actions of a first user of the terminal device, displays a first video of the real space as seen from the viewpoint of the first user, and, when it is determined that the first user has performed a predetermined action in the first video, causes the terminal device to function in the following way: display a second video of an animation of an avatar object based on the actions of a second user different from the first user, in combination with the first video."

[0011] Another aspect of the method can be described as "a method executed by at least one processor installed in a server device that executes computer-readable instructions, wherein the at least one processor executes the instructions to send a web page including a computer program to a terminal device, and the computer program, when executed by the terminal device that receives the web page, causes the terminal device to function in the following manner: operate based on the actions of a first user of the terminal device, display a first video of real space as seen from the perspective of the first user, and, when it is determined that the first user has performed a predetermined action in the first video, display a second video of animation of an avatar object based on the actions of a second user different from the first user, in combination with the first video." [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a communication system according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the hardware configuration of the terminal device 20 used in the communication system 1 shown in FIG. [Figure 3]FIG. 3 is a schematic diagram showing an example of a device that can be connected to the terminal device 20 or the like in order to generate motion data based on the first tracking method in the communication system 1 shown in FIG. [Figure 4A] FIG. 4A is a schematic diagram illustrating a first case regarding provision of moving images using the platform realized by the communication system 1 shown in FIG. [Figure 4B] FIG. 4B is a schematic diagram illustrating a first case regarding provision of moving images using the platform realized by the communication system 1 shown in FIG. [Figure 5A] FIG. 5A is a schematic diagram illustrating a second case regarding provision of moving images using the platform realized by the communication system 1 shown in FIG. [Figure 5B] FIG. 5B is a schematic diagram illustrating a second case regarding provision of moving images using the platform realized by the communication system 1 shown in FIG. [Figure 6A] FIG. 6A is a schematic diagram illustrating a third case regarding provision of moving images using the platform realized by the communication system 1 shown in FIG. [Figure 6B] FIG. 6B is a schematic diagram illustrating a third case regarding provision of moving images using the platform realized by the communication system 1 shown in FIG. [Figure 7] FIG. 7 is a diagram showing an example of a first distributor moving image displayed on a terminal device of the first distributor in the communication system 1 shown in FIG. [Figure 8] FIG. 8 is a diagram showing an example of a second distributor's moving image displayed on a terminal device of a first distributor in the communication system 1 shown in FIG. [Figure 9] FIG. 9 is a diagram showing another example of the first distributor video and the second distributor video displayed on the terminal device of the first distributor in the communication system 1 shown in FIG. [Figure 10] FIG. 10 is a diagram showing an example of a first viewer moving image displayed on the terminal device of viewer A in the communication system 1 shown in FIG. [Figure 11]FIG. 11 is a diagram showing another example of a first distributor moving image displayed on the terminal device of the first distributor in the communication system 1 shown in FIG. [Figure 12] FIG. 12 is a diagram showing another example of a second distributor's moving image displayed on the terminal device of the first distributor in the communication system 1 shown in FIG. [Figure 13] FIG. 13 is a flow diagram showing an example of the operation executed in the communication system 1 shown in FIG. [Figure 14] FIG. 14 is a diagram showing an example of a screen displayed when the terminal device in the communication system shown in FIG. 1 executes the "preparation mode." [Figure 15] FIG. 15 is a diagram showing an example of a first viewer video 700 displayed when the terminal device in the communication system shown in FIG. 1 is in the "viewing mode." [Figure 16] FIG. 16 is a diagram showing an example of a first viewer video 800 that is displayed when it is determined that a first predetermined action has been performed by a terminal device in the communication system shown in FIG. [Figure 17] Figure 17 is a diagram showing an example of a first viewer video 900 that is displayed when it is determined that a first predetermined action has been performed by a terminal device in the communication system shown in Figure 1 after the first viewer video 800 has been displayed. [Figure 18] Figure 18 is a diagram showing an example of a first viewer video 1000 that is displayed when it is determined that a first predetermined action has been performed by a terminal device in the communication system shown in Figure 1 after the first viewer video 800 has been displayed. [Figure 19] FIG. 19 is a diagram showing an example of a first viewer video 1100 that is displayed when it is determined that a second predetermined action has been performed by a terminal device in the communication system shown in FIG. [Figure 20] FIG. 20 is a diagram showing an example of a first viewer video 1200 that is displayed when it is determined that a second predetermined action has been performed by a terminal device in the communication system shown in FIG. [Figure 21]Figure 21 is a diagram showing an example of a first viewer video 1300 that is displayed when it is determined that a third predetermined action has been performed by a terminal device in the communication system shown in Figure 1 after the first viewer video 800 has been displayed. [Figure 22] Figure 22 is a diagram showing an example of a first viewer video 1400 that is displayed when it is determined that a fourth predetermined action has been performed by a terminal device in the communication system shown in Figure 1 after the first viewer video 800 has been displayed. [Figure 23] Figure 23 is a diagram showing an example of a first viewer video 1500 that is displayed when it is determined that a fifth predetermined action has been performed by a terminal device in the communication system shown in Figure 1 after the first viewer video 1400 has been displayed. [Figure 24] FIG. 16 is a diagram showing an example of a first viewer video 1600 that is displayed when it is determined that a first predetermined action has been performed by a terminal device in the communication system shown in FIG. [Figure 25] FIG. 25 is a diagram showing an example of a first viewer video 1700 in which the favorites list 520 is displayed. [Figure 26] FIG. 26 is a diagram showing an example of a first viewer video 1800 that is displayed when an icon 560 of each distributor is selected in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0013] This specification is described in terms of various representative embodiments, which are not intended to be limiting in any way. As used in this application, singular forms such as "a," "the," "said," "the," "the," "this," and "the" can include the plural unless clearly indicated as not being plural. Also, the term "comprises" can mean "comprises" or "has." Furthermore, the terms "coupled," "coupled," "coupled," "connected," or "connected" encompass mechanical, electrical, magnetic, and optical ways of coupling, connecting, or joining objects together, among other ways, and do not exclude the presence of intermediate elements between objects so coupled, coupled, coupled, coupled, connected, or connected.

[0014] The various systems, methods, and devices described herein should not be construed as limiting in any way. Indeed, the present disclosure is directed to all novel features and aspects of each of the various disclosed embodiments, combinations of these various embodiments with each other, and combinations of portions of these various embodiments with each other. The various systems, methods, and devices described herein are not limited to specific aspects, features, or combinations of such aspects with specific features, nor do the products and methods described herein require that one or more particular advantages be present or problems be solved. Furthermore, various features or aspects of the various embodiments described herein, or portions of such features or aspects, may be used in combination with each other.

[0015] Although the operations of some of the various methods disclosed herein may be described in a particular order for convenience, it should be understood that description in this manner encompasses rearranging the order of the operations unless a particular order is required by specific text below. For example, operations described in a sequence may, in some cases, be rearranged or performed simultaneously. Furthermore, for purposes of simplicity, the accompanying drawings may not show the various ways in which the various items and methods described herein may be used in conjunction with other items and methods. In addition, the specification may use terms such as "generate," "generate," "display," "receive," "evaluate," and "deliver." These terms are high-level descriptions of the various actual operations that may be performed. The actual operations corresponding to these terms may vary depending on the particular implementation and will be readily discernible by one of ordinary skill in the art having the benefit of the disclosure herein.

[0016] Any theories of operation, scientific principles, or other theoretical descriptions presented herein in connection with the devices or methods of the present disclosure are provided for the purpose of better understanding and are not intended to limit the scope of the technology, and the devices and methods in the appended claims are not limited to devices and methods that operate in a manner described by such theories of operation.

[0017] Any of the various methods disclosed herein may be implemented using computer-executable instructions stored on one or more computer-readable media (e.g., non-transitory computer-readable storage media, such as one or more optical media discs, volatile memory components, or non-volatile memory components), and executed on a computer. Here, the volatile memory components may include, for example, DRAM or SRAM. Also, the non-volatile memory components may include, for example, hard drives and solid-state drives (SSDs). Furthermore, the computer may include any computer available on the market, including, for example, smartphones and other mobile devices with computing hardware.

[0018] Any such computer-executable instructions for implementing the techniques disclosed herein, along with any data generated or used during the implementation of various embodiments disclosed herein, may be stored on one or more computer-readable media (e.g., non-transitory computer-readable storage media). Such computer-executable instructions may, for example, be part of a separate software application, or may be part of a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software may, for example, be executed on a single local computer (e.g., as a process running on any suitable commercially available computer) or in a networked environment (e.g., the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or other such network) using one or more networked computers.

[0019] For clarity, only certain selected aspects of various software-based implementations are described. Other details that are well known in the art are omitted. For example, the techniques disclosed herein are not limited to a particular computer language or program. For example, the techniques disclosed herein may be implemented by software written in C, C++, Java, or any other suitable programming language. Similarly, the techniques disclosed herein are not limited to a particular computer or type of hardware. Specific details of suitable computers and hardware are well known and need not be described in detail herein.

[0020] Moreover, any of the various software-based embodiments (e.g., including computer-executable instructions for causing a computer to perform any of the various methods disclosed herein) may be uploaded, downloaded, or remotely accessed by suitable communications means, including, for example, the Internet, the World Wide Web, an intranet, a software application, cable (including fiber optic cable), magnetic communication, electromagnetic communication (including RF, microwave, and infrared communication), electronic communication, or other such communications means.

[0021] The term "gift" used in this application has the same concept as the term "token." Therefore, the technology described in this application can be understood by replacing the term "gift" with the term "token."

[0022] Various embodiments of the present invention will be described below with reference to the accompanying drawings. Note that common components in the drawings are designated by the same reference numerals. It should also be noted that components depicted in one drawing may be omitted in another drawing for the sake of clarity. It should also be noted that the accompanying drawings are not necessarily drawn to scale.

[0023] In this specification, the terms "video distribution" by a certain distributor or "distributing a video" by a certain distributor may include, for example, at least one of the following exemplary aspects. A terminal device of a certain distributor, either alone or in collaboration with terminal devices of other distributors, transmits data (e.g., motion data) necessary for generating videos to terminal devices of multiple viewers via a server device, etc. A terminal device of a certain distributor, either alone or in collaboration with terminal devices of other distributors, transmits generated videos to each viewer's terminal device via a server device, etc.

[0024] 1. Overview In brief, the communication system disclosed in this application may perform the following operations.

[0025] In a typical example where a first broadcaster and a second broadcaster jointly broadcast a video to each viewer, the terminal device of the first broadcaster can generate first motion data based on the motion of the first broadcaster using a first tracking method and transmit it to the server device, and the terminal device of the second broadcaster can generate second motion data based on the motion of the second broadcaster using a second tracking method (different from the first tracking method) and transmit it to the server device.

[0026] Here, the first tracking method may include at least a method of generating motion data indicating the coordinates and angles of a device worn on the streamer's head and at least one controller worn or held on a part of the streamer's body other than the streamer's head. The device worn on the streamer's head may include, but is not limited to, an HMD, smart glasses (e.g., virtual reality (VR) glasses, augmented reality (AR) glasses, and mixed reality (MR) glasses), etc. The at least one controller may include, but is not limited to, a hand controller held in the streamer's right hand, a hand controller held in the streamer's left hand, a controller worn on the streamer's waist, etc.

[0027] In addition, the second tracking method may include at least a method (such as a face tracking method) that generates motion data indicating the depth of each of multiple parts of the broadcaster's face and the coordinates and angle of the broadcaster's head.

[0028] The server device can transmit the first motion data and the second motion data to a plurality of terminal devices of viewers, a terminal device of the first broadcaster, and a terminal device of the second broadcaster.

[0029] Each of the plurality of terminal devices, including the terminal devices of each viewer, the terminal device of the first broadcaster, and the terminal device of the second broadcaster, can identify the first motion data and the second motion data received from the server device. Then, each of the plurality of terminal devices can generate and display a motion image including an animation (moving image) of the avatar of the first broadcaster based on the first motion data and an animation of the avatar of the second broadcaster based on the second motion data.

[0030] On the other hand, in an example where a first distributor solely provides video to each viewer, the terminal device of this first distributor can transmit first movement data generated using a first tracking method based on the movement of this first distributor to the server device.

[0031] The server device can transmit the first motion data to terminal devices of a plurality of viewers and a terminal device of the first distributor.

[0032] Each of the plurality of terminal devices, including the terminal devices of the viewers and the terminal device of the first broadcaster, can identify the first motion data received from the server device. Then, each of the plurality of terminal devices can generate and display a video including an animation of the avatar of the first broadcaster based on the first motion data.

[0033] On the other hand, in an example where a second distributor solely provides video to each viewer, the terminal device of the second distributor can transmit second movement data generated using a second tracking method based on the movement of the second distributor to the server device.

[0034] The server device can transmit the second motion data to the terminal devices of the plurality of viewers and the terminal device of the second distributor.

[0035] Each of the plurality of terminal devices, including the terminal devices of the viewers and the terminal device of the second broadcaster, can identify the second motion data received from the server device. Then, each of the plurality of terminal devices can generate and display a video including an animation of the avatar of the second broadcaster based on the second motion data.

[0036] In the above examples, a method (client rendering method) is adopted in which video generation is performed by each of the multiple terminal devices. However, a method (server rendering method) in which such video generation is performed by the server device (or another server device that can be connected to the server device) and the video thus generated is transmitted to and displayed on the multiple terminal devices may also be adopted. Alternatively, a method (browser rendering method) in which video generation is performed by each of the multiple terminal devices receiving a web page from the server device (or another server device that can be connected to the server device) and executing a computer program included in the web page may also be adopted.

[0037] 2. Overall configuration of the communication system Next, the configuration of a communication system that executes the above-described operations will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of a communication system according to an embodiment.

[0038] As shown in Fig. 1, the communication system 1 can include at least one server device 30, a plurality of terminal devices 20 connectable to the communication network 10, and at least one server device 30 connectable to the communication network 10. Fig. 1 shows an example in which the plurality of terminal devices 20 include three terminal devices 20A, 20B, and 20C, but the plurality of terminal devices 20 can include any number (two or more) of terminal devices 20. Similarly, Fig. 1 shows an example in which the at least one server device 30 includes three server devices 30A, 30B, and 30C, but the at least one server device 30 can also include any number (one or more) of server devices 30.

[0039] The multiple terminal devices 20 may include at least one distributor's terminal device 20 and at least one viewer's terminal device 20. Each of the at least one distributor's terminal device 20 is capable of generating and transmitting motion data based on either the first tracking method or the second tracking method, and each of the at least one viewer's terminal device 20 is also capable of generating and transmitting motion data based on either the first tracking method or the second tracking method.

[0040] Furthermore, the multiple terminal devices 20 may be any information processing device, including, but not limited to, an HMD, VR glasses, AR glasses, MR glasses, a smartphone, a personal computer, a tablet, a mobile information terminal, a mobile phone, and combinations thereof.

[0041] The at least one server device 30 may be an information processing device capable of providing various services to each of the multiple terminal devices 20 that access the server device 30. The at least one server device 30 may be any information processing device, and may include, but is not limited to, a personal computer, a supercomputer, a workstation, a mainframe, and combinations thereof.

[0042] Furthermore, at least one server device 30 may include a web server device 30 that transmits a web page including a computer program to each of a plurality of terminal devices 20 that access this server device 30.

[0043] Furthermore, at least one server device 30 may be connectable to one another via the communication network 10.

[0044] The communication network 10 may include, without limitation, a cellular network, a wireless network, a landline network, the Internet, an intranet, a local area network (LAN), a wide area network (WAN), and / or an Ethernet network, where the wireless network may include RF connections via, for example, Bluetooth, WiFi (such as IEEE 802.11a / b / n), WiMax, cellular, satellite, laser, and / or infrared.

[0045] 3. Hardware Configuration of Terminal Device 20 Next, the configuration of the terminal device 20 used in the above-described communication system 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of the terminal device 20 used in the communication system 1 shown in Fig. 1 (note that in Fig. 2, the reference numerals in parentheses are written in relation to the server device 30, as will be described later).

[0046] 2, each terminal device 20 mainly includes a central processing unit 21, a main memory device 22, an input / output interface device 23, an input device 24, an auxiliary memory device 25, and an output device 26. These devices are connected to each other by a data bus and / or a control bus.

[0047] The central processing unit 21 is called a "CPU" and can perform operations on instructions and data stored in the main memory 22 and store the results of the operations in the main memory 22. Furthermore, the central processing unit 21 can control an input device 24, an auxiliary memory device 25, an output device 26, etc. via an input / output interface device 23. The terminal device 20 can include one or more such central processing units 21.

[0048] The main memory device 22 is referred to as "memory" and can store instructions and data received from the input device 24, the auxiliary memory device 25, and the communication network 10 (such as the server device 30) via the input / output interface device 23, as well as the results of calculations by the central processing unit 21. The main memory device 22 can include, but is not limited to, computer-readable media such as volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), EEPROM, flash memory), and storage (e.g., hard disk drive (HDD), solid-state drive (SSD), magnetic tape, optical media). As will be readily understood, the term "computer-readable recording medium" can include media for data storage such as memory and storage, as well as transmission media such as modulated data signals or transitory signals.

[0049] The auxiliary storage device 25 is a storage device having a larger capacity than the main storage device 22. The auxiliary storage device 25 can store instructions and data (computer programs) that constitute an operating system, a web browser application, a specific application for distributing and / or receiving video, etc. Furthermore, the auxiliary storage device 25 can transmit these instructions and data (computer programs) to the main storage device 22 via the input / output interface device 23 under the control of the central processing unit 21. The auxiliary storage device 25 can include, but is not limited to, a magnetic disk device and / or an optical disk device, etc.

[0050] The input device 24 is a device that inputs data from the outside and may include, but is not limited to, a touch panel, a button, a keyboard, a mouse, and / or a sensor. The sensor may include, but is not limited to, one or more cameras 24a and / or one or more microphones 24b.

[0051] The output device 26 may include, but is not limited to, a display device, a touch panel, and / or a printer device.

[0052] In a first aspect, the terminal device 20 may be a terminal device 20 that generates motion data based on a first tracking method. In this case, the terminal device 20 (typically a personal computer, etc.) may have (or be connected to) a device (such as an HMD or VR glasses) that functions as the input device 24 and the output device 26 and is worn on the user's head, and at least one controller that functions as the input device 24 and is worn or held on a part of the user's body other than the head. The at least one controller may include, for example, but is not limited to, a right-hand controller that may be worn or held on the user's right hand and a left-hand controller that may be worn or held on the user's left hand. Furthermore, when the terminal device 20 generates motion data based on the first tracking method, the terminal device 20 may not have a controller.

[0053] Fig. 3 is a schematic diagram showing an example of a device connectable to the terminal device 20 or the like to generate motion data based on the first tracking method in the communication system 1 shown in Fig. 1. The terminal device 20 that generates motion data based on the first tracking method may include (or be connected to) an HMD (or smart glasses or the like) 100 worn on the user's head, a right hand controller 102 that can be held or worn on a part of the user's body other than the head (here, the user's right hand), and a left hand controller 104 that can be held or worn on a part of the user's body other than the head (here, the user's left hand), as shown in Fig. 3.

[0054] The HMD 100 may include a base 100A and a fixing unit 100B for fixing the base 100A to the user's head (not shown). The base 100A may include a display unit (not shown) arranged facing the user's eyes, a sensor unit (not shown) that detects the coordinates and angle of the user's head and generates motion data indicating these, a camera unit (not shown) arranged near the user's eyes and acquires video data (or image data), and a communication unit (not shown) that communicates various data (including motion data, operation data, etc.) between the display unit, the sensor unit, the camera unit, the right hand controller 102, the left hand controller 104, and the terminal device 20 (the input / output interface device 23). By providing the HMD 100 with a display unit and a camera unit, virtual objects (e.g., a broadcaster's avatar) can be combined in real space, and the HMD 100 can function as an MR terminal device. Note that the display unit may be configured to transmit ambient light to the user's eyes without providing a camera unit. Furthermore, by providing a display unit in the HMD 100, it is possible to display a virtual space, and the HMD 100 can function as a VR terminal device. Details of the terminal device functioning as an MR terminal device or a VR terminal device will be described later.

[0055] The sensor unit may also include an eye-tracking sensor (not shown) that faces the user's eyeball so as to directly detect the movement of the user's eyeball. The eye-tracking sensor may be, for example, a gaze detection sensor configured to irradiate near-infrared light into the iris and detect the reflected light. Such an eye-tracking sensor can generate operation data indicating the movement of the user's gaze. A user wearing such an HMD 100 can move an object (such as a pointer) displayed on the display unit and / or select an object displayed on the display unit by moving their eyeball. Furthermore, a user wearing the HMD 100 can move the direction of the camera unit by moving their eyeball, thereby acquiring images (or video data) corresponding to the movement of their eyeball.

[0056] The fixing part 100B may include a first fixing part 100B1 that extends in a substantially U-shape and has both ends fixed to the base part 100A, and a second fixing part 100B2 that has one end fixed to the base part 100A and the other end fixed to the first fixing part 100B1. The first fixing part 100B1 and the second fixing part 100B2 surround the user's head and are biased toward the base part 100A, allowing the HMD 100 to be worn on the user's head.

[0057] The right hand controller 102 and the left hand controller 104 may have housings 102A and 104A, respectively, that are symmetrical to each other. The right hand controller 102 is provided in the housing 102A and may include, for example, a joystick 102B that can be operated by a user's finger and at least one button (here, for example, buttons 102C, 102D, and 102E) that can be pressed by the user's finger. Furthermore, the right hand controller 102 may include a sensor unit (not shown) provided inside the housing 102A and configured to detect the coordinates and angle of the user's right hand and generate motion data indicating these, and to generate operation data indicating operations performed by the user on each of the joystick 102B and the buttons 102C, 102D, and 102E, and a communication unit (not shown) that communicates various data (including motion data and operation data) with the HMD 100.

[0058] Similarly, the left hand controller 104 is provided in the housing 104A and can include, for example, a joystick 104B that can be operated by the user's finger, and at least one button (here, for example, buttons 104C, 104D, and 104E) that can be pressed by the user's finger. Furthermore, the left hand controller 104 can include a sensor unit (not shown) that is provided inside the housing 104A and configured to detect the coordinates and angle of the user's right hand, generate motion data indicating these, and generate operation data indicating operations performed by the user on each of the joystick 104B and the buttons 104C, 104D, and 104E, and a communication unit (not shown) that communicates various data (including motion data and operation data) between the left hand controller 104 and the HMD 100.

[0059] The communication unit of the HMD 100 can transmit the motion data and / or operation data generated by the sensor unit of the HMD 100, the motion data and / or operation data generated by the right hand controller 102, and the motion data and / or operation data generated by the left hand controller 104 to the terminal device 20, etc.

[0060] 2, in a second aspect, the terminal device 20 may be a terminal device 20 that generates motion data based on a second tracking method. In this case, the terminal device 20 (typically a smartphone, tablet, personal computer, or the like) may have (or be connected to) at least one camera 24a (and at least one microphone 24b) functioning as the input device 24 and a touch panel device functioning as the output device 26.

[0061] In such a hardware configuration, the central processing unit 21 can sequentially load instructions and data (computer programs) constituting the specific application stored in the auxiliary storage unit 25 into the main storage unit 22 and perform operations on the loaded instructions and data. This allows the central processing unit 21 to control the output unit 26 via the input / output interface unit 23, and also to transmit and receive various data to and from other devices (e.g., the server device 30 and / or other terminal devices 20) via the input / output interface unit 23 and the communication network 10.

[0062] As a result, the terminal device 20 can execute various operations related to the distribution, generation and / or display of moving images (including operations described with reference to Figures 4A to 4B, Figures 5A to 5B, Figures 6A to 6B, Figure 13, etc.) by executing the installed specific applications, etc.

[0063] It should be noted that the terminal device 20 may include one or more microprocessors and / or a graphics processing unit (GPU) instead of or in addition to the central processing unit 21.

[0064] 4. Hardware Configuration of Server Device 30 Next, the configuration of the server device 30 used in the above-described communication system 1 will be described with reference to Fig. 2. The hardware configuration of each server device 30 may be the same as the hardware configuration of each of the above-described terminal devices 20. Therefore, reference numerals for the components of each server device 30 are shown in parentheses in Fig. 2.

[0065] 2, each server device 30 can include a central processing unit 31, a main memory device 32, an input / output interface device 33, an input device 34, an auxiliary memory device 35, and an output device 36. These devices are connected to each other by a data bus and / or a control bus.

[0066] The central processing unit 31, main memory 32, input / output interface unit 33, input device 34, auxiliary memory 35, and output device 36 may be substantially the same as the central processing unit 21, main memory 22, input / output interface unit 23, input device 24, auxiliary memory 25, and output device 26 included in each of the terminal devices 20 described above, respectively.

[0067] In such a hardware configuration, the central processing unit 31 can sequentially load instructions and data (computer programs) constituting the specific application stored in the auxiliary storage device 35 into the main storage device 32 and perform operations on the loaded instructions and data. This allows the central processing unit 31 to control the output device 36 via the input / output interface device 33, and also to transmit and receive various data to and from other devices (e.g., other server devices 30 and / or terminal devices 20) via the input / output interface device 33 and the communication network 10.

[0068] As a result, the server device 30 can execute various operations related to the distribution, generation and / or display of moving images (including operations described with reference to Figures 4A to 4B, Figures 5A to 5B, Figures 6A to 6B, Figure 13, etc.) by executing the installed specific applications, etc.

[0069] Note that the server device 30 may include one or more microprocessors and / or a graphics processing unit (GPU) instead of or in addition to the central processing unit 31.

[0070] The server device 30 can also function as a web server by executing the specific application installed therein. The server device 30 can transmit a web page including a computer program to each terminal device 20 (and / or other server devices 30) that accesses the server device 30. By executing the computer program included in the web page, the terminal device 20 (and / or other server devices 30) can also perform various operations related to the distribution, generation, and / or display of moving images (including the operations described with reference to FIGS. 4A to 4B, 5A to 5B, 6A to 6B, 13, etc.).

[0071] 5. Platform Next, we will explain an example of a platform realized by the communication system 1. This platform makes it possible for each distributor's terminal device and each viewer's terminal device to appropriately display videos, regardless of whether each distributor uses a terminal device 20 that generates and transmits motion data based on the first tracking method or a terminal device 20 that generates and transmits motion data based on the second tracking method.

[0072] The operations performed on this platform will be explained using the following three cases as examples. For ease of explanation, the terminal device 20 that generates and transmits operations based on the first tracking method will be referred to as an "HMD terminal device" (including VR glasses, AR glasses, and MR glasses), and the terminal device 20 that generates and transmits operations based on the second tracking method will be referred to as a "smartphone terminal device." However, even a smartphone or tablet can become an "HMD terminal device" by using multiple tracking devices (for example, the devices shown in FIG. 3) connected to it.

[0073] Case 1 (Figs. 4A-4B): A first broadcaster using an HMD terminal and a second broadcaster using a smartphone terminal jointly provide a video to each viewer. After this, one viewer (viewer A) uses an HMD terminal to jointly provide a video to each viewer with the first and second broadcasters. Second case (Fig. 5A-5B): The first distributor using the HMD terminal device provides the video to each viewer independently. After this, viewer A uses the HMD terminal device to provide the video to each viewer in cooperation with the first distributor. Third case (Fig. 6A-6B): The second distributor using a smartphone terminal device provides the video to each viewer independently. After this, viewer A uses an HMD terminal device to provide the video to each viewer in cooperation with the second distributor.

[0074] 5-1. Case 1 First, the first case will be described with reference to Figures 4A and 4B. Figures 4A and 4B are schematic diagrams illustrating the first case regarding provision of moving images using a platform realized by the communication system 1 shown in Figure 1. For convenience, the communication network 10 intervening between each terminal device 20 and the server device 30 is omitted in Figures 4A and 4B.

[0075] (1) About Figure 4A 4A, a first distributor using an HMD terminal device 20A and a second distributor using a smartphone terminal device 20B are jointly providing video to the terminal devices 20 of multiple viewers, including an HMD terminal device 20C of viewer A. However, for the sake of simplicity, only the terminal device 20C of viewer A, which is an HMD terminal device, is shown in FIGS. 4A and 4B as the terminal devices 20 of the multiple viewers.

[0076] The terminal device 20A of the first broadcaster can generate motion data (first motion data) based on the motion of the first broadcaster using a first tracking method at unit time intervals or at any timing. The length of the unit time may be fixed or variable. This first motion data may include motion data indicating coordinates and angles (orientations) (e.g., associated with timestamps) generated by each of the tracking devices included in the multiple tracking devices attached to the body of the first broadcaster.

[0077] For example, if the terminal device 20A uses the tracking device illustrated in FIG. 3, the first operation data may include, but is not limited to, the following exemplary data. Motion data generated by the HMD 100, indicating the coordinates and angle of the head (the sensor unit mounted on the HMD 100 has a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor, and is therefore capable of generating motion data indicating the coordinates and angle of the head). Motion data indicating the coordinates and angle of the right hand generated by the right hand controller 102 (the above-mentioned sensor unit mounted on the right hand controller 102 has a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor, and is therefore able to generate motion data indicating the coordinates and angle of the right hand.) Motion data indicating the coordinates and angle of the left hand, generated by the left hand controller 104 (the above-mentioned sensor unit mounted on the left hand controller 104 has a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor, and is therefore able to generate motion data indicating the coordinates and angle of the left hand.) Operation data generated by the right hand controller 102, which indicates the content of an operation performed on a joystick or the like with the right hand Operation data generated by the left hand controller 104, which indicates the content of an operation performed on a joystick or the like with the left hand

[0078] Furthermore, the terminal device 20A of the first distributor can also generate voice data (first voice data) based on the speech of the first distributor (for example, using the microphone 24b).

[0079] The terminal device 20A can continue to transmit the first action data (and optionally the first voice data) generated in this way to the server device 30 at unit time intervals or at any timing.

[0080] On the other hand, the terminal device 20B of the second broadcaster can generate motion data (second motion data) based on the motion of the second broadcaster using a second tracking method at unit time intervals or at any timing. The length of the unit time may be fixed or variable. This second motion data may include motion data indicating the depth of each of multiple facial features and the coordinates and angle of the head (e.g., associated with a timestamp), generated by image processing using at least one sensor (e.g., camera 24a). For example, a dot projector emits an infrared laser forming a dot pattern onto the user's face, and the near-infrared camera 24a captures the infrared dots projected and reflected from the user's face, generating an image of the captured infrared dots. The terminal device 20B can compare an image of a dot pattern emitted by a pre-registered dot projector with an image captured by the near-infrared camera, and calculate the depth of each point (feature point) (e.g., each of 51 points) (feature point) in both images (feature point) (the distance between each point and the near-infrared camera) using the positional deviation of each point (feature point). The terminal device 20B can generate motion data indicating the depth calculated in this way. Furthermore, the terminal device 20B has a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor, and can generate motion data indicating the coordinates and angle of the head.

[0081] Furthermore, the terminal device 20B can also generate voice data (second voice data) based on the speech of the second distributor (for example, using the microphone 24b).

[0082] The terminal device 20B can continue to transmit the second action data (and optionally the second voice data) generated in this way to the server device 30 at unit time intervals or at any timing.

[0083] The server device 30 can continue to transmit "one set of action data" including the first action data (and optionally the first voice data) from the terminal device 20A and the second action data (and optionally the second voice data) from the terminal device 20B to each of the terminal device 20A, the terminal device 20B, and the terminal device 20 of viewer A (as well as to the terminal devices 20 of the other viewers not shown). In this sense, it can be said that "one set of action data" is data that is commonly transmitted to the terminal device 20A, the terminal device 20B, and the terminal device 20 of viewer A (as well as to the terminal devices 20 of the other viewers not shown).

[0084] The server device 30 can also add, for example, the following exemplary data to the "one set of operation data" and transmit it. (A) Data identifying the terminal device 20 and / or sender that transmitted the motion data (and audio data) generated by each of the terminal devices 20 (here, terminal devices 20A and 20B) of multiple distributors. (B) Data indicating whether the motion data (and audio data) generated by each of the terminal devices 20 (here, terminal devices 20A and 20B) of multiple distributors was generated using the first tracking method or the second tracking method.

[0085] Each terminal device 20 that receives such "one set of motion data" can identify how many distributors' generated motion data (and audio data) are included in the "one set of motion data" by referring to data (A). Furthermore, each terminal device 20 can identify which part of the "one set of motion data" contains the motion data (and audio data) generated by the terminal device of each distributor.

[0086] Furthermore, by referring to data (B), each terminal device 20 can identify whether the motion data (and audio data) generated by each broadcaster's terminal device was generated using the first tracking method or the second tracking method. Each terminal device 20 recognizes in advance what data the motion data (and audio data) generated using each tracking method includes. For example, each terminal device 20 recognizes in advance that the motion data generated using the first tracking method includes data indicating the coordinates and angles of the corresponding parts generated by each tracking device (and operation data indicating the content of the operation performed by the broadcaster on the joystick, button, etc.). Similarly, each terminal device 20 recognizes in advance that the motion data generated using the second tracking method includes data indicating the depth of each of multiple parts of the face and the coordinates and angle of the head. Therefore, by referring to data (B), each terminal device 20 can recognize and use the content of the motion data generated using either tracking method by each broadcaster's terminal device 20.

[0087] In order to enable each terminal device 20 to extract and utilize the action data generated by each distributor's terminal device from the "one set of action data" received, the server device 30 can add any other data to any position of the "one set of action data."

[0088] Terminal device 20A, terminal device 20B, and terminal device 20C (terminal devices 20 of each of the other viewers) can generate a video based on the received "one set of motion data." In the following description, each terminal device 20 can receive and store, from server device 30, in advance or at any time, drawing data for drawing an avatar used by each broadcaster in association with avatar identification data that identifies the avatar. Therefore, by identifying the broadcaster, each terminal device 20 can identify the avatar used by that broadcaster and generate and display animation of the avatar.

[0089] First, attention is focused on the terminal device 20A. The terminal device 20A can generate a moving image including an animation of a first broadcaster's avatar (first avatar) based on the first motion data and an animation of a second broadcaster's avatar (second avatar) based on the second motion data. Specifically, because the terminal device 20A is an HMD terminal device, it can generate two types of moving images: a first broadcaster moving image and a second broadcaster moving image. Note that the terminal device 20A can extract the first motion data and the second motion data from a set of motion data received from the server device 30 using the above-described method, and use the extracted data to generate a moving image.

[0090] The first broadcaster video may be a video in which a first avatar and a second avatar are arranged in a first virtual space (e.g., a three-dimensional virtual space) and drawn (expressed) from the "first broadcaster's viewpoint" when the terminal device 20A functions as a VR terminal device. The first broadcaster video may be a video in which a first avatar and a second avatar are arranged in real space as seen from the "first broadcaster's viewpoint" when the terminal device 20A functions as an MR terminal device. The second broadcaster video may be a video in which a first avatar and a second avatar are arranged in the first virtual space and drawn (expressed) from a certain viewpoint in the first virtual space (e.g., a virtual camera arranged in the first virtual space) when the terminal device 20A functions as a VR terminal device. Furthermore, this second video may be a video drawn (expressed) from a "viewpoint common to all viewers." That is, the certain viewpoint may be a viewpoint common to all viewers. The following description will be given of a case in which the terminal device 20A functions as a VR terminal device and generates the first broadcaster video and the second broadcaster video.

[0091] In another embodiment, the second distributor video may be a video drawn (expressed) from a "common viewpoint for each viewer and each distributor." In other words, the certain viewpoint may be a common viewpoint for each viewer and each distributor.

[0092] Fig. 7 is a diagram showing an example of a first distributor video displayed on a terminal device of a first distributor in the communication system 1 shown in Fig. 1. As shown in Fig. 7, the first distributor video 200 displayed on the terminal device 20A of the first distributor may include a first avatar 210 of the first distributor that moves according to the first motion data and a second avatar 220 of the second distributor that moves according to the second motion data. Here, the first distributor video 200 does not include an avatar (third avatar) of viewer A.

[0093] First distributor video 200 may be a video depicting the first virtual space as seen from the viewpoint of first avatar 210 placed in the first virtual space (i.e., the viewpoint of the first distributor). The current position of first avatar 210 in the first virtual space and the angle at which first avatar 210 faces may be determined and changed according to first action data included in one set of video data received from server device 30 (or first action data generated and temporarily stored by terminal device 20A).

[0094] For example, when the first broadcaster turns his / her head 15 degrees to the right (or left), the terminal device 20A can include data indicating the angle of the head rotated 15 degrees to the right (or left) in the action data. As a result, the terminal device 20A updates the angle of the head of the first avatar 210, so that the angle at which the first avatar 210 faces is rotated 15 degrees to the right (or left) in the first broadcaster video 200. Furthermore, when the first broadcaster raises the right hand controller 102 and the left hand controller 104 up to his / her chest, the terminal device 20A can include data indicating the changed coordinates of the right hand and the left hand in the action data. As a result, the terminal device 20A updates the coordinates of the right hand and the left hand of the first avatar 210, so that the right hand 210a and the left hand 210b of the first avatar 210 are drawn raised in the first broadcaster video 200, as illustrated in FIG. 7. Furthermore, when the first broadcaster presses button 102E on the right hand controller 102 (or button 104E on the left hand controller 104), the terminal device 20A can include data indicating the coordinates of the head that has been moved forward (or updated) in the video data. As a result, the terminal device 20A updates the coordinates at which the first avatar 210 is located, and the viewpoint of the first avatar 210 moves forward (or backward) in the first broadcaster video 200. Accordingly, the second avatar 220 is drawn larger (or smaller) in the first broadcaster video 200.

[0095] In the first distributor video 200, just as the first avatar 210 moves according to the first motion data, the second avatar 220 moves according to the second motion data. However, because the second distributor's terminal device 20B is a smartphone terminal device, the second motion data generated by the terminal device 20B is motion data generated based on the second tracking method (motion data indicating the depth of each of multiple facial features and the coordinates and angle of the head). Therefore, in the first distributor video 200, the head of the second avatar 220 moves (for example, shakes its head from side to side) based on the second motion data (for example, data indicating the coordinates and angle of the head). Furthermore, the facial expression of the second avatar 220 changes based on the second motion data (data indicating the depth of each of multiple facial features). Furthermore, the direction in which the second avatar 220 faces changes based on the second motion data (data indicating the coordinates and angle of the head). However, the position of the second avatar 220 does not change.

[0096] Figure 8 is a diagram showing an example of a second distributor video displayed on a terminal device of a first distributor in the communication system 1 shown in Figure 1. As illustrated in Figure 8, the second distributor video 300 includes a first avatar 210 of the first distributor that moves according to the first motion data and a second avatar 220 of the second distributor that moves according to the second motion data, but does not include an avatar (third avatar) of viewer A.

[0097] The second distributor video 300 is a video depicting the first virtual space from a common viewpoint for all viewers (including viewer A). The common viewpoint here may be a viewpoint from any point in the first virtual space (such as a virtual camera installed at this point). For simplicity, FIG. 8 shows an example in which both the first avatar 210 and the second avatar 220 are depicted as viewed from the front. However, depending on the position of the viewpoint in the first virtual space, the first avatar 210 and / or the second avatar 220 may face sideways or backwards in the second distributor video 300. In one embodiment, the second distributor video 300 may be generated so that both the first avatar 210 and the second avatar 220 face forward, regardless of the position of the viewpoint in the first virtual space.

[0098] Because the first motion data is generated according to a first tracking method, the position of the first avatar 210 in the first virtual space, which moves based on this first motion data, may change, and the direction in which the first avatar 210 faces may also change. Therefore, in the second distributor video 300, the position of the first avatar 210 may change, and the direction in which the first avatar 210 faces may also change. In contrast, because the second motion data is generated according to a second tracking method, the coordinates in the first virtual space of the second avatar 220, which moves based on this second motion data, do not change. Therefore, in the second distributor video 300, the position of the second avatar 220 does not change (however, in the second distributor video 300, the facial expression of the second avatar 220, etc., may change based on the second motion data).

[0099] For this reason, in one embodiment, as illustrated in FIG. 8, the second avatar 220, whose position in the first virtual space is fixed, may be placed in the front row, and the first avatar 210, whose position in the first virtual space changes, may be placed in the back row behind the front row. Furthermore, in one embodiment, as illustrated in FIG. 8, the first avatar 210 placed in the back row may be placed higher than the second avatar 220 placed in the front row. In this way, by placing multiple avatars in rows (front row and back row) that correspond to their characteristics, each of the multiple avatars can be made easy to see for each user (first broadcaster, second broadcaster, and each viewer). Note that in another embodiment, it is also possible to place the first avatar 210 in the front row and the second avatar 220 in the back row, inversely to the above.

[0100] In one embodiment, the terminal device 20A can display the first broadcaster video 200 described above with reference to FIG. 7 and the second broadcaster video 300 described above with reference to FIG. 8 in different regions on a display unit (such as the HMD 100) that is the output device 26. For example, the terminal device 20A can display the first broadcaster video 200 in a specific region, such as the right, left, top, or bottom, of a specific region included in a display region formed by the display unit, and display the second broadcaster video 300 in the remaining region of the specific region. In another preferred embodiment, the terminal device 20A can also display the second broadcaster video 300 within the region where the first broadcaster video 200 is displayed, as illustrated in FIG. 9.

[0101] FIG. 9 is a diagram showing another example of a first broadcaster video and a second broadcaster video displayed by the terminal device of the first broadcaster in the communication system 1 shown in FIG. 1. As shown in FIG. 9, the terminal device 20A can combine the second broadcaster video 300 with the first broadcaster video 200 and display it, for example, within the area where the first broadcaster video 200 is displayed. Specifically, the terminal device 20A can display the second broadcaster video 300 movably in conjunction with a part (here, the left hand 210b) of the first avatar 210 included in the first broadcaster video 200. In this case, when the first broadcaster moves his or her left hand holding the left hand controller 104, the left hand 210b of the first avatar 210 included in the first broadcaster video 200 moves accordingly. The second broadcaster video 300 held (drawn as if held) by the left hand 210b can also move in conjunction with the left hand 210b. In one embodiment, the second broadcaster video 300 may be displayed as a pseudo terminal device (pseudo terminal device 350) displayed in the first broadcaster video 200.

[0102] Returning to FIG. 4A, attention is now focused on terminal device 20B. Terminal device 20B can generate a video including an animation of the first broadcaster's avatar (first avatar) based on the first motion data and an animation of the second broadcaster's avatar (second avatar) based on the second motion data. Specifically, since terminal device 20A is a smartphone terminal device, it can generate a second broadcaster video. Note that terminal device 20B can extract the first motion data and the second motion data from a set of motion data received from server device 30 using the above-described method, and use the extracted data to generate a video.

[0103] As described above, the second distributor video may be a video in which a first avatar and a second avatar are placed in the first virtual space and drawn (expressed) as viewed from a certain viewpoint in the first virtual space (for example, a virtual camera placed in the first virtual space). Furthermore, this second distributor video may be a video drawn (expressed) as viewed from a "viewpoint common to all viewers." Here, the "one set of motion data" that the terminal device 20B receives from the server device 30 is the same as the "one set of motion data" that the terminal device 20A receives from the server device 30. Furthermore, the method by which the terminal device 20B generates the second distributor video is the same as the method by which the terminal device 20A generates the second distributor video described above. As a result, the second distributor video generated by the terminal device 20B may be substantially the same as the second distributor video 300 (see FIG. 8) generated by the terminal device 20A. Therefore, the terminal device 20B can display the second distributor video 300, as illustrated in FIG. 8, on the output device 26, such as a touch panel or a display.

[0104] Referring again to FIG. 4A, attention is now focused on terminal device 20C. The terminal device 20C can generate a moving image including an animation of an avatar (first avatar) of a first broadcaster based on the first motion data and an animation of an avatar (second avatar) of a second broadcaster based on the second motion data. Specifically, because the terminal device 20C is an HMD terminal device, it can generate two types of moving images: a first viewer moving image and a second viewer moving image. Note that the terminal device 20C can extract the first motion data and the second motion data from a set of motion data received from the server device 30 using the above-described method, and use the extracted data to generate a moving image.

[0105] The first viewer video may be a video in which the first avatar and the second avatar are arranged in real space as seen from "viewpoint of viewer A" when the terminal device 20C functions as an MR terminal device. The second viewer video may be a video in which the first avatar and the second avatar are arranged in a first virtual space as seen from a certain viewpoint in the first virtual space (for example, a virtual camera arranged in the first virtual space) when the terminal device 20C functions as a VR terminal device. Furthermore, this second viewer video may be a video that is drawn (expressed) as seen from a "viewpoint common to all viewers." The second viewer video may be the same video as the second distributor video.

[0106] Fig. 10 is a diagram showing an example of a first viewer video displayed on the terminal device of viewer A in the communication system 1 shown in Fig. 1. As shown in Fig. 10, the first viewer video 400 displayed on the terminal device 20C of viewer A can include a first avatar 210 of the first broadcaster that moves according to the first action data, and a second avatar 220 of the second broadcaster that moves according to the second action data. The display conditions for the first avatar 210 of the first broadcaster and the second avatar 220 of the second broadcaster will be described later.

[0107] Because the terminal device 20C is an HMD terminal device, it can generate movement data (third movement data) using the first tracking method based on the movement of viewer A, who is the user of the terminal device 20C. Note that viewer A may or may not use the right hand controller 102 and the left hand controller 104 to generate the third movement data. In the example described below, the right hand controller 102 and the left hand controller 104 are omitted from the viewer video.

[0108] The first viewer video 400 may be a video in which animation of a first avatar and animation of a second avatar are arranged in video data of real space seen from "viewpoint of viewer A." In this example, the video data of real space seen from "viewpoint of viewer A" is video data acquired by a camera unit. In this example, the camera unit is provided near both of the user's eyes, so that video data of real space corresponding to the movement of viewer A's head can be acquired. In FIG. 10, viewer A's hands 470 (viewer A's right hand 470a and left hand 470b) and sofa 480 are displayed in the first viewer video 400 as objects in real space.

[0109] Furthermore, the animation of the first avatar and the second avatar that are rendered may be determined and changed according to the third action data. For example, the video data of the real space changes when viewer A moves forward (or backward) from a position in the real space. As the video data of the real space changes, first avatar 210 and second avatar 220 are rendered larger (or smaller) in first viewer video 400.

[0110] In the first viewer video 400, the first avatar 210 moves in accordance with the first movement data. Because the terminal device 20A of the first broadcaster is an HMD terminal device, the first movement data generated by the terminal device 20A is movement data generated based on the first tracking method. Therefore, in the first viewer video 400, the position of the first avatar 210, the direction in which the first avatar 210 faces, and the like may change based on the first movement data.

[0111] Additionally, in the first viewer video 400, the second avatar 220 moves in accordance with the second movement data. However, because the terminal device 20B of the second broadcaster is a smartphone terminal device, the second movement data generated by the terminal device 20B is movement data generated based on the second tracking method (movement data indicating the depth of each of multiple parts of the face and the coordinates and angle of the head). Therefore, in the first viewer video 400, the head of the second avatar 220 moves (for example, shaking its head from side to side) based on the second movement data, but the position of the second avatar 220 does not change.

[0112] As described above, the second viewer video may be a video in which a first avatar and a second avatar are placed in the first virtual space and drawn (expressed) as viewed from a certain viewpoint in the first virtual space. Furthermore, this second video may be a video drawn (expressed) as viewed from a "viewpoint common to all viewers." Here, the "one set of motion data" that the terminal device 20C receives from the server device 30 is the same as the "one set of motion data" that the terminal devices 20A and 20B receive from the server device 30. Furthermore, the method by which the terminal device 20C generates the second viewer video is the same as the method by which the terminal devices 20A and 20B generate the second distributor video described above. As a result, the second viewer video generated by the terminal device 20C may be substantially the same as the second distributor video 300 (see FIG. 8 ) generated by the terminal devices 20A and 20B. Therefore, the terminal device 20C can display the second distributor video 300, as illustrated in FIG. 8 , on the output device 26, such as the HMD 100.

[0113] Similarly to terminal device 20A, terminal device 20C can also display a combination of first viewer video 400 and second viewer video. The technique described above with reference to Fig. 9 in relation to terminal device 20A can also be applied to first viewer video 400 and second viewer video displayed by terminal device 20C. For example, pseudo terminal device 350 can be displayed in first viewer video 400, similar to Fig. 9.

[0114] (2) Regarding Figure 4B FIG. 4B shows a transition from the situation shown in FIG. 4A to a situation in which viewer A, who is a user of terminal device 20C, acts as a third distributor and provides a video in collaboration with the first and second distributors.

[0115] 4B differs from FIG. 4A in that terminal device 20C, which is an HMD terminal device, transmits third movement data generated using the first tracking method based on the movement of the third broadcaster to server device 30. As a result, server device 30 continues to transmit one set of movement data including the first movement data, the second movement data, and the third movement data to terminal device 20A, terminal device 20B, terminal device 20C, and each viewer's terminal device 20 (not shown).

[0116] Attention is now focused on the terminal device 20A. The terminal device 20A can generate a moving image including an animation of a first broadcaster's avatar (first avatar) based on the first motion data, an animation of a second broadcaster's avatar (second avatar) based on the second motion data, and an animation of a third broadcaster's avatar (third avatar) based on the third motion data. Specifically, because the terminal device 20A is an HMD terminal device, it can generate two types of moving images: a first broadcaster moving image and a second broadcaster moving image. Note that the terminal device 20A can extract the first motion data, the second motion data, and the third motion data from one set of motion data received from the server device 30 using the above-described method, and use the extracted data to generate a moving image.

[0117] Figure 11 is a diagram showing another example of a first distributor video displayed on a terminal device of a first distributor in the communication system 1 shown in Figure 1. As shown in Figure 11, a first distributor video 200A displayed on terminal device 20A of the first distributor includes a first avatar 210 of the first distributor that moves according to the first motion data, a second avatar 220 of the second distributor that moves according to the second motion data, and a third avatar 450 that moves according to the third motion data.

[0118] The method of generating or displaying the first avatar 210 and the second avatar 220 in the first distributor video 200A is the same as that in the first distributor video 200, and therefore will not be described here.

[0119] In the first distributor video 200A, the third avatar 450 moves according to the third motion data. Because the third distributor's terminal device 20C is an HMD terminal device, the third motion data generated by the terminal device 20C is motion data generated based on the first tracking method. Therefore, in the first distributor video 200A, the position of the third avatar 450, the direction in which the third avatar 450 faces, and the like may change based on the third motion data.

[0120] Figure 12 is a diagram showing another example of a second distributor video displayed on a terminal device of a first distributor in the communication system 1 shown in Figure 1. As illustrated in Figure 12, the second distributor video 300A includes a first avatar 210 of the first distributor that operates according to the first motion data, a second avatar 220 of the second distributor that operates according to the second motion data, and a third avatar 450 of the third distributor that operates according to the third motion data.

[0121] The second distributor video 300A differs from the second distributor video 300 shown in FIG. 8 in that the second distributor video 300A includes a third avatar 450 (other aspects are as described above in relation to the second distributor video 300).

[0122] Because the third motion data is generated according to the first tracking method, the position of third avatar 450, which moves based on this third motion data, in the first virtual space may change, and the direction in which third avatar 450 faces may also change. Therefore, in second distributor video 300A, the position of third avatar 450 may change, and the direction in which third avatar 450 faces may also change.

[0123] Additionally, the third avatar 450 may preferably be placed in the back row together with the first avatar 210 for the reasons described above in relation to the second broadcaster video 300.

[0124] 5-2. Case 2 In the second case, the first distributor using the HMD terminal device provides video to each viewer independently. After this, viewer A uses the HMD terminal device to provide video to each viewer in cooperation with the first distributor.

[0125] The second case corresponds to the first case (FIGS. 4A and 4B) described above, excluding the second distributor, its terminal device 20B, and the second behavior data. That is, FIGS. 5A and 5B correspond to FIGS. 4A and 4B, respectively, excluding the second distributor, its terminal device 20B, and the second behavior data. Therefore, to avoid redundant explanation, FIGS. 5A and 5B only show the data transmitted between devices (first behavior data to fourth behavior data), and will not be described again.

[0126] 5-3. Case 3 In the third case, the second distributor using a smartphone terminal device provides video to each viewer independently. After this, viewer A uses an HMD terminal device to provide video to each viewer in collaboration with the second distributor.

[0127] The third case corresponds to the first case (FIGS. 4A and 4B) described above, excluding the first distributor, its terminal device 20A, and the first operational data. That is, FIGS. 6A and 6B correspond to FIGS. 4A and 4B, excluding the first distributor, its terminal device 20A, and the first operational data. Therefore, to avoid redundant explanation, FIGS. 6A and 6B only show the data transmitted between devices, and their explanation will be omitted.

[0128] 5-4. Modifications In the various embodiments described above, the server device 30 transmits all of the motion data received from the terminal device 20 of each broadcaster as one set of motion data to the terminal device 20 of each broadcaster and to the terminal device 20 of each viewer. That is, in the various embodiments described above, the server device 30 transmits certain motion data, including the certain motion data in one set of motion data, to the terminal device of the broadcaster that transmitted the certain motion data.

[0129] In this case, as the total number of distributors jointly distributing videos increases, the amount of motion data each distributor's terminal device 20 receives from the server device 30, i.e., the amount of motion data the server device 30 transmits to each distributor's terminal device 20, increases. As a result, a problem may arise in that the communication network becomes congested, making it difficult for each distributor's terminal device 20 to smoothly provide videos to each viewer's terminal device 20 via the server device 30. To address this problem, in another embodiment, the server device 30 may be configured to not transmit certain motion data to a terminal device of a distributor that has transmitted the certain motion data. This reduces the amount of motion data each distributor's terminal device 20 receives from the server device 30, i.e., the amount of motion data the server device 30 transmits to each distributor's terminal device 20. Therefore, videos can be more smoothly provided from each distributor's terminal device 20 to each viewer's terminal device 20 via the server device 30. In this case, the terminal device of the distributor can generate a video using the certain motion data it generated. 4A and 4B, the server device 30 can transmit the first action data only to the terminal device 20B and each viewer's terminal device 20, but not to the terminal device 20A, and similarly transmit the second action data only to the terminal device 20A and each viewer's terminal device 20, but not to the terminal device 20B. Furthermore, in the configuration shown in Fig. 4B, the server device 30 can prevent the third action data from being transmitted to the terminal device 20C.

[0130] Furthermore, although not explicitly mentioned in the various embodiments described above because it is self-evident, each broadcaster's terminal device and each viewer's terminal device can acquire audio data generated by each broadcaster's terminal device that is included in or combined with the "set of motion data" received from the server device. Using such audio data, each broadcaster's terminal device and each viewer's terminal device can play the first broadcaster / viewer video and / or the second broadcaster / viewer video incorporating the audio emitted by each broadcaster, or can play the audio emitted by each broadcaster while displaying the first broadcaster / viewer video and / or the second broadcaster / viewer video.

[0131] Furthermore, in the various examples described above, we have focused on a system (client rendering system) in which each terminal device 20 executes operations related to the generation of the first distributor / viewer video and the second distributor / viewer video. However, it is also possible to adopt a system (server rendering system) in which the operations related to the generation of videos, which have been described in each of FIGS. 4A to 6B as being executed by each terminal device 20, are executed solely by one server device 30 or shared among multiple server devices 30. In this case, the one or more server devices 30 execute various operations related to the generation of videos on behalf of each terminal device 20, thereby generating the first viewer video and / or the second viewer video and transmitting them to the terminal device 20C. As described above, the server device 30 receives motion data from the terminal device 20 of each distributor, and can generate various videos using the motion data on behalf of each terminal device 20. The server device 30 may also receive video data acquired by the camera unit of the terminal device 20C. However, the server device 30 does not necessarily receive motion data from each viewer's terminal device 20. Therefore, when the server rendering method is adopted, the server device 30 may receive the motion data from the terminal device of each viewer at each unit time or at any timing.

[0132] In addition, in the various examples described above, the explanation has been focused on a method (client rendering method) in which each terminal device 20 executes operations related to the generation of the first distributor / viewer video and the second distributor / viewer video, etc. However, each terminal device 20 may also employ a method (browser rendering method) in which each terminal device 20 receives a web page from the server device 30 or the web server device 30 via a web browser and executes a computer program included in the web page to execute operations related to the generation of the first distributor / viewer video and the second distributor / viewer video, etc.

[0133] According to the platform described above, regardless of whether a broadcaster's terminal device uses the first tracking method or the second tracking method to generate motion data, each terminal device can generate or display video according to the tracking method adopted by the broadcaster's terminal device. Furthermore, each terminal device can generate or display one or both of the available videos of the first broadcaster and viewer and the second broadcaster and viewer, depending on whether the terminal device is an HMD terminal device or a smartphone terminal device. This allows broadcasters to provide videos to viewers, etc., using tracking devices worn on their bodies.

[0134] 6. Operations Performed by Communication System 1 6-1. Basic operation Next, a specific example of the operation performed in the communication system 1 using the above-mentioned platform will be described, focusing on the first viewer video displayed by the terminal device 20. In the following, to avoid redundant explanation, the matters explained in relation to the above-mentioned platform will be omitted.

[0135] Fig. 13 is a flow diagram showing an example of the operation executed in the communication system 1 shown in Fig. 1. First, in step (hereinafter referred to as "ST") 502, a user (here, "viewer A") operates the terminal device 20C to cause the terminal device 20C to start and execute a specific application for receiving the provision of a video service. Note that the operation described below also applies to any other terminal device (HMD terminal device) 20.

[0136] Next, in ST504, the terminal device 20C executing the specific application can execute a "preparation mode." This "preparation mode" is a mode in which the terminal device 20C prepares to receive or distribute video when it is not receiving video provided by another terminal device 20 or distributing video to another terminal device 20. The terminal device 20C can store mode identification data indicating which mode is currently being executed. The terminal device 20C can store the mode identification data indicating the "preparation mode" from when it transitions to the "preparation mode" until it ends the "preparation mode." At any timing, the terminal device 20C can determine which mode is being executed by referring to the mode identification data.

[0137] When the terminal device 20C is in the "preparation mode", it can display a screen (home screen) such as that shown in FIG.

[0138] Fig. 14 is a diagram showing an example of a screen displayed when a terminal device in the communication system shown in Fig. 1 executes the "preparation mode." As shown in Fig. 14, terminal device 20C, which is an HMD terminal device, can display, on the home screen, a first viewer video 600 of the real space as seen from the viewpoint of viewer A. As described above in "5. Platform," first viewer video 600 includes video data acquired by the camera unit of terminal device 20C. In first viewer video 600 in Fig. 14, a sofa 480 is displayed as an object in the real space.

[0139] Furthermore, the terminal device 20C can display an input object 610 in combination with the first viewer video 600. The input object 610 can function as a user interface that enables viewer A to input operations. Specifically, the input object 610 can include multiple icons, such as an icon 620 for switching the terminal device 20C to a "viewing mode," an icon 622 for switching the terminal device 20C to a "solo distribution mode," and an icon 624 for switching the terminal device 20C to a "joint distribution mode" ("collaboration distribution mode"). In FIG. 14 , the input object 610 is displayed at the center of the first viewer video 600.

[0140] Since the first viewer video 600 is a video of the real space seen from the viewpoint of viewer A, viewer A can display a part of his / her body by moving a part of his / her body (for example, the viewer's hand). Therefore, when a part of viewer A's body overlaps with an icon, the mode corresponding to that icon can be selected. This allows viewer A to operate the displayed terminal device in a pseudo manner, as if he / she were operating the terminal device by touching the touch panel of the terminal device, such as a smartphone.

[0141] Note that the case where a part of viewer A's body overlaps with an icon may be the case where a specific part of viewer A overlaps with an icon. The specific part of viewer A may be, for example, viewer A's fingertips. To prevent malfunction, it may be determined that a part of viewer A's body overlaps with an icon if the part of viewer A's body overlaps with the icon for a certain period of time.

[0142] The terminal device 20C may also display a pointer (not shown) associated with the input object 610 that operates in accordance with the viewer A's operation of a tracking device worn or held by the viewer A, for example, the right-hand controller 102 or the left-hand controller 104. The terminal device 20C can render the pointer as a laser beam flowing from bottom to top on the paper. The terminal device 20C can also render a pointer trajectory. The viewer A can move the pointer on the surface of the input object 610 by moving the right-hand controller 102 or the left-hand controller 104. The viewer A can select a mode corresponding to one of the icons 620, 622, or 624 by positioning the pointer on the icon and pressing a button on the right-hand controller 102 or the left-hand controller 104.

[0143] Terminal device 20C can also display second viewer video 640 in input object 610. Second viewer video 640 can display third avatar 450, which moves based on motion data representing the motion of viewer A, in the first virtual space, as seen from the front.

[0144] Furthermore, the terminal device 20C can display such an input object 610 as a pseudo-represented terminal device (pseudo terminal device), as exemplified in Fig. 14. The pseudo terminal device may include, as one aspect, a pseudo-represented pseudo smartphone (pseudo smart phone).

[0145] Furthermore, in one embodiment, terminal device 20C can display first viewer video 600 and input object 610 in different regions of a display unit (such as HMD 100) that is output device 26. For example, terminal device 20C can display first viewer video 600 in a specific region, such as the right, left, top, or bottom, of a specific region included in a display region formed by the display unit, and display input object 610 in the remaining region of the specific region. In another preferred embodiment, terminal device 20A can display input object 610 within the region where first viewer video 600 is displayed, as illustrated in FIG. 14. In this case, the position where input object 610 is displayed may be set by viewer A as appropriate.

[0146] Returning to FIG. 13, next, in ST506, terminal device 20C can determine whether or not any mode (here, for example, "viewing mode," "single distribution mode," or "shared distribution mode") has been selected by viewer A while executing the "preparation mode." If terminal device 20C determines that none of the modes has been selected, it can continue to execute ST504. On the other hand, if terminal device 20C determines that any of the modes has been selected, it can proceed to a step corresponding to the selected mode. In this example, a case will be described where it has been determined in ST506 that the "viewing mode" has been selected.

[0147] Next, in ST508, terminal device 20C transitions to the "viewing mode." In the "viewing mode," terminal device 20C displays first viewer video 700.

[0148] FIG. 15 is a diagram showing an example of a first viewer video 700 displayed by a terminal device in the communication system shown in FIG. 1 when the terminal device is in "viewing mode." As shown in FIG. 15, terminal device 20C, which is an HMD terminal device, can display a first viewer video 700 of the real space seen from the viewpoint of viewer A. Like first viewer video 600 in FIG. 14, first viewer video 700 includes video data acquired by the camera unit of terminal device 20C. In FIG. 15, first viewer video 700 displays viewer A's hands 470 (viewer A's right hand 470a and left hand 470b) and sofa 480. As shown in FIG. 15, first viewer video 700 does not include animation of the broadcaster's avatar.

[0149] Returning to FIG. 13 again, next, in ST510, it is determined whether viewer A has performed a predetermined action in the first viewer video 700. The predetermined action is a specific user action. The predetermined action may be an action based on the user's hand movement. As an example, the predetermined action may be a user clapping action. The user clapping action may be a user clapping action in which the user performs multiple consecutive actions of putting both hands together and separating both hands. The predetermined action may be a user touching an object displayed in the first viewer video 700. An example of a user touching an object displayed in the first viewer video 700 is a user touching the sofa 480 in FIG. 15. An example of a user touching an object displayed in the first viewer video 700 is a user touching the ground in FIG. 15. The predetermined action may be a user making a specific hand sign (for example, a thumbs-up gesture). The predetermined action may be a predetermined action whose execution can be determined. The predetermined action is not limited to these examples. In this example, the predetermined action will be described as a user clapping their hands, and this action will be referred to as a first predetermined action. The first predetermined action is an appearance action that causes an avatar of the broadcaster to appear.

[0150] The terminal device 20C may determine whether a predetermined action has been executed. When the server rendering method is adopted, the server device 30 may determine whether a predetermined action has been executed. An example of determining whether the terminal device 20C has executed a first predetermined action will be described below.

[0151] The terminal device 20C may determine whether a predetermined action has been performed using the first viewer video 700. In one aspect, the terminal device 20C may acquire position coordinates of an object in real space from the first viewer video 700 by image recognition or the like. For example, the position coordinates of the right hand 470a may be a collection of position information of an object recognized as the right hand 470a. The terminal device 20C may determine whether a predetermined action has been performed using the position coordinates of the object in real space. For example, in the case of a user clapping their hands, if the set of position coordinates of the right hand 470a and the set of position coordinates of the left hand 470b overlap, it is determined that the user has performed an action of putting their hands together, and if the set of position coordinates of the right hand 470a and the set of position coordinates of the left hand 470b separate, it is determined that the user has performed an action of separating their hands. Then, when the user performs the actions of putting both hands together and separating both hands multiple times in succession (i.e., when the user claps his / her hands "clap clap..."), it is determined that the user has performed the action of clapping his / her hands (first predetermined action). Alternatively, when the user performs the action of putting both hands together at a speed faster than a predetermined speed and then separates his / her hands (i.e., when the user claps his / her hands vigorously "clap!"), it may be determined that the user has performed the action of clapping his / her hands (first predetermined action).

[0152] In another aspect, terminal device 20C may recognize objects in real space from first viewer video 700 by image recognition or the like. For example, terminal device 20C recognizes right hand 470a and left hand 470b in first viewer video 700. Then, if it is recognized that the recognized right hand 470a and left hand 470b are in contact, it may be determined that the user has performed an action of putting their hands together, and if it is recognized that the recognized right hand 470a and left hand 470b are not in contact, it may be determined that the user has performed an action of separating their hands. Alternatively, if it is simply recognized that the recognized right hand 470a and left hand 470b overlap on the first viewer video 700, it may be determined that the user has performed an action of putting their hands together, and if it is recognized that the recognized right hand 470a and left hand 470b separate on the first viewer video 700, it may be determined that the user has performed an action of separating their hands.

[0153] As yet another aspect, it may be determined that the user has performed an action of putting both hands together or an action of separating both hands based on information acquired from a motion sensor (for example, a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor) included in the terminal device 20C. In this case, as an example, the user wears or holds a controller having a motion sensor.

[0154] Then, in ST510, if it is determined whether viewer A has performed a predetermined action in first viewer video 700, an animation of the distributor's avatar is displayed in first viewer video 700 in ST512.

[0155] Fig. 16 is a diagram showing an example of a first viewer video 800 that is displayed when it is determined that a first predetermined action has been performed by a terminal device in the communication system shown in Fig. 1. In this example, a second avatar 220 of a second broadcaster is displayed in the first viewer video 800.

[0156] The position where the second avatar 220 of the second broadcaster is displayed is a position that is a first predetermined distance away from the position of viewer A. In other words, the second avatar 220 of the second broadcaster is displayed at a position that is the first predetermined distance away from the position of viewer A in real space. The first predetermined distance is, for example, 1 meter. The position where the second avatar 220 of the second broadcaster is displayed may be a position that is the first predetermined distance away from the position of viewer A in the direction in which viewer A clapped. The direction in which viewer A clapped may be the direction in which both hands of viewer A are displayed. In this example, viewer A claps toward the center of the first viewer video 800, and the second avatar 220 of the second broadcaster is displayed in the center of the first viewer video 800. Alternatively, the second avatar 220 of the second broadcaster may be displayed in the center of the first viewer video 800 simply when viewer A claps, regardless of the direction in which viewer A clapped.

[0157] In this example, when it is determined that the first predetermined action has been performed, the second avatar 220 of the second broadcaster is displayed, but the example of the broadcaster avatar that is displayed is not limited to this. When it is determined that the first predetermined action has been performed, the first avatar 210 of the first broadcaster may be displayed. When it is determined that the first predetermined action has been performed, an avatar selected randomly from among the broadcasters currently broadcasting may be displayed.

[0158] Furthermore, the broadcaster to be displayed when it is determined that the first predetermined action has been performed may be selected or determined based on viewer A's viewing history. For example, the broadcaster may be determined based on a list preset by viewer A, a list related to viewer A's viewing history, and a list related to viewer A's viewing history. More specifically, for example, when it is determined that the first predetermined action has been performed, an avatar may be displayed randomly from among broadcasters currently broadcasting and registered in the favorites list. When it is determined that the first predetermined action has been performed, an avatar may be displayed from among broadcasters currently broadcasting and registered in the favorites list in the order of their top ranking (for example, in the favorites list in FIG. 25 described later, avatars are displayed in the order of "Mr. U," "Mr. V," and "Mr. W"). When it is determined that the first predetermined action has been performed, an avatar may be displayed from among broadcasters currently broadcasting and registered in the favorites list based on a predetermined priority. When it is determined that the first predetermined action has been performed, an avatar may be displayed from among broadcasters currently broadcasting and registered in the favorites list who have co-broadcasted the most times with viewer A. Furthermore, when it is determined that the first predetermined action has been performed, the avatar of the currently broadcasting broadcaster who has been viewed the most by viewer A may be displayed. Furthermore, when it is determined that the first predetermined action has been performed, the avatar of the currently broadcasting broadcaster whose time period most overlaps with the time period in which viewer A previously viewed the broadcast may be displayed. Furthermore, when it is determined that the first predetermined action has been performed, the avatar of the currently broadcasting broadcaster whose previous viewing time by viewer A is less than a predetermined value may not be displayed. For example, if viewer A has only viewed a particular broadcaster for a few seconds (e.g., 3 seconds) in the past, it may be determined that viewer A is not interested in that broadcaster, and the avatar of that broadcaster may not be displayed (in this case, the avatar of a broadcaster other than that broadcaster may be displayed).

[0159] Furthermore, the broadcaster displayed when it is determined that the first predetermined action has been performed may be selected or determined based on the viewing history of another viewer (referred to as viewer B) other than viewer A. In this case, the broadcaster displayed when it is determined that the first predetermined action has been performed may include a broadcaster that viewer A has never viewed. For example, when it is determined that the first predetermined action has been performed, the avatar of a broadcaster who is currently broadcasting and has been viewed the most by viewer B may be displayed. Viewer B may be determined based on viewer A's viewing history. Viewer B may be a viewer whose viewing history correlates with viewer A's. The viewing histories of viewer B and viewer A being correlated may mean that the broadcasters that viewer B has viewed in the past partially overlap with the broadcasters that viewer A has viewed in the past. The viewing histories of viewer B and viewer A being correlated may mean that the favorite list of broadcasters of viewer B and viewer A partially overlap with the favorite list of broadcasters of viewer A. Viewer B may also be determined randomly.

[0160] As described above, when it is determined that viewer A has performed the first predetermined action, the broadcaster's avatar is displayed in the first viewer video 800. Therefore, the broadcaster's avatar can be projected into real space at any time desired by viewer A, allowing viewer A to enjoy the MR space. Furthermore, because the broadcaster's avatar moves based on the motion data, as described below, viewer A can feel immersed in the MR space. Furthermore, after ST512, the viewing mode continues.

[0161] 17 is a diagram showing an example of a first viewer video 900 that is displayed when it is determined that a first predetermined action has been performed by a terminal device in the communication system shown in Fig. 1 after first viewer video 800 has been displayed. In this example, first viewer video 900 displays first avatar 210 of the first broadcaster in addition to second avatar 220 of the second broadcaster.

[0162] The position where the first avatar 210 of the first broadcaster is displayed is a position that is a second predetermined distance away from the position of viewer A. In other words, the first avatar 210 of the first broadcaster is displayed at a position that is the second predetermined distance away from the position of viewer A in real space. The second predetermined distance is, for example, 1 meter. The position where the first avatar 210 of the first broadcaster is displayed may be a position that is the second predetermined distance away from the position of viewer A in the direction in which viewer A clapped. The second predetermined distance may be the same as or different from the first predetermined distance. Alternatively, the first avatar 210 of the first broadcaster may be displayed in the center of the first viewer video 900 simply when viewer A clapped, regardless of the direction in which viewer A clapped.

[0163] In this example, the distance between the position of the first avatar 210 of the first broadcaster and the position of the second avatar 220 of the second broadcaster is equal to or greater than a predetermined value. The predetermined value is, for example, a value that prevents the first avatar 210 of the first broadcaster and the second avatar 220 of the second broadcaster from partially overlapping each other. The predetermined value may be set by viewer A as appropriate. A case in which the distance between the position of the first avatar 210 of the first broadcaster and the position of the second avatar 220 of the second broadcaster is equal to or less than the predetermined value will be described with reference to FIG. 18.

[0164] As described above, each time it is determined that viewer A has performed a first predetermined action, the avatar of the determined number of times distribution follower is displayed in the first viewer video 900. Therefore, viewer A can project the avatars of multiple broadcasters into real space, allowing viewer A to enjoy the MR space. Note that if there is no broadcaster that can be displayed when it is determined that the first predetermined action has been performed (for example, if the second broadcaster is displayed in the first viewer video 800 and no other broadcasters are broadcasting), the other broadcasters do not need to be displayed.

[0165] 18 is a diagram showing an example of a first viewer video 1000 that is displayed when it is determined that a first predetermined action has been performed by a terminal device in the communication system shown in Fig. 1 after first viewer video 800 has been displayed. In this example, first avatar 210 of the first broadcaster is displayed in first viewer video 1000 instead of second avatar 220 of the second broadcaster.

[0166] When displaying the first avatar 210 of the first broadcaster, if the distance between the position of the first avatar 210 of the first broadcaster and the position of the second avatar 220 of the second broadcaster is equal to or less than a predetermined value, the first avatar 210 of the first broadcaster and the second avatar 220 of the second broadcaster may partially overlap. In this case, a video in which the first avatar 210 of the first broadcaster and the second avatar 220 of the second broadcaster overlap is displayed in the first viewer video 1000, which makes viewer A feel uncomfortable and makes it difficult for him or her to achieve a sense of immersion in the MR space.

[0167] In this example, when displaying the first avatar 210 of the first broadcaster, if the distance between the position of the first avatar 210 of the first broadcaster and the position of the second avatar 220 of the second broadcaster is less than a predetermined value, the video of the second avatar 220 of the second broadcaster is deleted, so that the first avatar 210 of the first broadcaster and the second avatar 220 of the second broadcaster do not partially overlap. This prevents the sense of immersion that viewer A experiences in the MR space from being lost. Furthermore, viewer A can freely change the broadcaster to be displayed, allowing viewer A to enjoy the MR space.

[0168] As described above, because the first motion data is generated according to the first tracking method, the position of the first avatar 210 in the first virtual space may change. Therefore, even if the distance between the position of the first avatar 210 of the first broadcaster and the position of the second avatar 220 of the second broadcaster is equal to or greater than a predetermined value when it is determined that viewer A has performed the first predetermined action (i.e., when the broadcaster's avatar starts to be displayed), the distance between the position of the first avatar 210 of the first broadcaster and the position of the second avatar 220 of the second broadcaster may become equal to or less than the predetermined value after a certain period of time has elapsed. In this case, the movement of the first avatar 210 may be restricted so that the distance between the position of the first avatar 210 and the position of the second avatar 220 of the second broadcaster becomes equal to or greater than the predetermined value. For example, the movement of the first avatar 210 may be invalidated if the distance between the position of the first avatar 210 and the position of the second avatar 220 of the second broadcaster becomes equal to or less than the predetermined value. Furthermore, if the distance between the position of the first avatar 210 of the first distributor and the position of the second avatar 220 of the second distributor is equal to or less than a predetermined value, the video of the avatar of one of the distributors may be deleted.

[0169] 6-2. Another Example 19 is a diagram showing an example of first viewer video 1100 that is displayed when it is determined that a second predetermined action has been performed by a terminal device in the communication system shown in FIG. 1. The second predetermined action is an appearance action that causes the broadcaster's avatar to appear, and may be an action different from the first predetermined action. In this example, the second predetermined action is a gesture in which viewer A touches an object displayed in first viewer video 1100. In FIG. 19, the object is sofa 480 that exists in the real space where viewer A is located.

[0170] The second avatar 220 of the second broadcaster is displayed at the target position. In this example, the second avatar 220 of the second broadcaster is displayed at the position of the sofa 480. In FIG. 19, the second avatar 220 of the second broadcaster is displayed at the position of the sofa 480. As shown in FIG. 19, the second avatar 220 of the second broadcaster may be displayed above the sofa 480. The second avatar 220 may also be displayed so as to overlap the sofa 480.

[0171] When it is determined that viewer A has performed the second predetermined action, the broadcaster's avatar is displayed in the first viewer video 1100. Therefore, the broadcaster's avatar can be projected into real space at any timing of viewer A's choosing, allowing viewer A to enjoy the MR space. Furthermore, because the broadcaster's avatar moves based on motion data as described below, viewer A can feel immersed in the MR space. Note that, as in the above-described embodiment, if the second predetermined action is performed multiple times, multiple broadcaster avatars may be displayed. When a new broadcaster's avatar is displayed, if the distance between the position of the newly displayed broadcaster's avatar and the position of the previously displayed avatar is less than a predetermined value, the video of the previously displayed avatar may be deleted.

[0172] Fig. 20 is a diagram showing an example of first viewer video 1200 that is displayed when it is determined that a second predetermined action has been performed by a terminal device in the communication system shown in Fig. 1. In this example, the display of second avatar 220 of the second broadcaster in first viewer video 1200 is different from the display of second avatar 220 of the second broadcaster in first viewer video 1100 in Fig. 19.

[0173] In this example, when the second avatar 220 of the second broadcaster is displayed, the display of the second avatar 220 of the second broadcaster is changed based on the object in which the second avatar 220 of the second broadcaster is located in the first viewer video 1200. In this example, since the second avatar 220 of the second broadcaster is displayed at the position of the sofa 480, the second avatar 220 of the second broadcaster is displayed as if making a motion of sitting on the sofa 480. When the second avatar 220 of the second broadcaster is displayed at the position of the sofa 480, the second avatar 220 of the second broadcaster may be displayed as if making a motion of lying down on the sofa 480. Furthermore, when the second avatar 220 of the second broadcaster is displayed on the ground, the second avatar 220 of the second broadcaster may be displayed standing upright. When the second avatar 220 of the second broadcaster is displayed on the ground, the second avatar 220 of the second broadcaster may be displayed as if making a specific pose. The display of the second avatar 220 of the second broadcaster is changed based on the object where the second avatar 220 of the second broadcaster is located, so viewer A can feel immersed in the MR space.

[0174] The avatar whose display changes based on the object in which the avatar is located may be limited to the avatar of the broadcaster on the smartphone terminal device (e.g., the second avatar 220 of the second broadcaster). The avatar of the broadcaster on the smartphone terminal device can be smoothly changed because the position of the avatar does not change.

[0175] 21 is a diagram showing an example of first viewer video 1300 that is displayed when it is determined that a third predetermined action has been performed by the terminal device in the communication system shown in FIG. 1 after first viewer video 800 has been displayed. In this example, a heart-shaped gift 490 is displayed in first viewer video 1300 in addition to second avatar 220 of the second broadcaster. In FIG. 21, the video of second avatar 220 of the second broadcaster and gift 490 are displayed in combination.

[0176] When it is determined that the third predetermined action has been performed, a gift 490 may be displayed. The third predetermined action is a gift action for granting a gift. Here, the third predetermined action may be an action different from the first predetermined action and the second predetermined action described above. In this example, the third predetermined action is an action of pointing a finger heart shape with a finger toward a specific avatar. The third predetermined action may also be an action of simply forming a finger heart. If the third predetermined action is simply an action of forming a finger heart, a gift may be granted to all avatars displayed in the first viewer video 1300. Furthermore, if the third predetermined action is simply an action of forming a finger heart, a gift may be granted to an avatar displayed near the center of the first viewer video 1300. Since a gift is granted when it is determined that the third predetermined action has been performed, viewer A can enjoy watching the video even more. Furthermore, gift information regarding the gift granted by viewer A may be transmitted to the broadcaster's terminal device and reflected in the video displayed on the broadcaster's terminal device. Therefore, the broadcaster and the viewer can communicate through gifts.

[0177] The gift information may include, for example, the following data: (A) Data identifying the terminal device 20 and / or the giver (sender) of the gift given by the viewer's terminal device 20 (here, terminal device 20C). (B) Data identifying the terminal device 20 and / or recipient that received (was given) the gift given by the viewer's terminal device 20 (here, terminal device 20C). (C) Data indicating whether the gift given by viewer A's terminal device 20C was generated according to the first gifting method or the second gifting method.

[0178] The first gifting method is a gifting method that does not consume the parameters of the giver (sender). As an example, the gift displayed by the third predetermined action is a gift generated by the first gifting method. The second gifting method is a gifting method that consumes the parameters of the giver (sender). When a gift is given by the second gifting method, for example, the giver's (sender's) in-game currency or coins are consumed. As an example, the gifts displayed by the fourth and fifth predetermined actions described below are gifts generated by the second gifting method. In this case, after the gift is displayed by the fourth predetermined action, a confirmation dialog may be displayed when the fifth predetermined action is performed to confirm that the giver's (sender's) parameters will be consumed.

[0179] 22 is a diagram showing an example of first viewer video 1400 that is displayed when it is determined that a fourth predetermined action has been performed by the terminal device in the communication system shown in FIG. 1 after first viewer video 800 has been displayed. In this example, cake-shaped gift 514 is displayed in first viewer video 1300 in addition to second avatar 220 of the second broadcaster. In FIG. 22, gift 514 is displayed superimposed on hand 470.

[0180] When it is determined that a fourth predetermined action has been performed, gift 514 may be displayed. Here, the fourth predetermined action may be an action different from the first, second, and third predetermined actions described above. In this example, the fourth predetermined action is an action of raising both hands upward. When it is determined that the fourth predetermined action has been performed, gift 514 is displayed in conjunction with hand 470, so that viewer A is shown holding a cake in hand 470 in first viewer video 1400. The fourth predetermined action may also be an action of holding a glass, a wine bottle, etc. in one hand. When the fourth predetermined action is an action of holding a glass, a wine bottle, etc. in one hand and this action is performed, a glass-shaped or wine bottle-shaped gift is displayed. The fourth predetermined action may also be an action of moving both hands apart a predetermined distance from a state in which they are touching. When the fourth predetermined action is an action of moving both hands apart a predetermined distance from a state in which they are touching and this action is performed, a cake tower-shaped gift is displayed in viewer A's hand.

[0181] 23 is a diagram showing an example of a first viewer video 1500 that is displayed when it is determined that a fifth predetermined action has been performed by a terminal device in the communication system shown in FIG. 1 after first viewer video 1400 has been displayed. In this example, a video 240 about the second avatar of the second broadcaster and gift 514 is displayed in first viewer video 1000 instead of second avatar 220 of the second broadcaster.

[0182] When it is determined that the fifth predetermined action has been performed, a video 240 of the second avatar of the second broadcaster and the gift 514 may be displayed. The fifth predetermined action is a gift action of giving a gift. Here, the fifth predetermined action may be an action different from the first, second, third, and fourth predetermined actions described above. In this example, the fifth predetermined action is an action of throwing the gift 514.

[0183] Video 240 relating to the second avatar of the second broadcaster and gift 514 may be a video showing the second avatar holding gift 514. If gift 514 is an accessory such as a ring, video 240 relating to the second avatar of the second broadcaster and gift 514 may be a video showing the second avatar wearing gift 514. When it is determined that the fifth predetermined action has been performed, video 240 relating to the second avatar of the second broadcaster and gift 514 is displayed, allowing viewer A to enjoy the MR space.

[0184] The second broadcaster's screen may display the second broadcaster's second avatar and a video related to gift 514, similar to viewer A's screen. Alternatively, the second broadcaster's screen may display only the second broadcaster's second avatar, without displaying a video related to the second broadcaster's second avatar and gift 514. When a gift is given to the second broadcaster, the second broadcaster's screen may display a notification regarding the gift given by viewer A to the second broadcaster, such as "Viewer A has sent you a cake gift." In this case, the second broadcaster may be able to select whether to continue displaying the second broadcaster's second avatar or to display a video related to the second broadcaster's second avatar and gift 514 (for example, a video of the second avatar holding a cake).

[0185] Figure 24 is a diagram showing an example of a first viewer video 1600 that is displayed when it is determined that a first predetermined action has been performed by a terminal device in the communication system shown in Figure 1. In this example, the first predetermined distance is longer than the first predetermined distance in the first viewer video 800 of Figure 16. Therefore, the second avatar 220 of the second broadcaster in the first viewer video 1600 of Figure 24 is drawn smaller than in the first viewer video 800 of Figure 16.

[0186] In this example, not only is the second avatar 220 of the second broadcaster displayed, but when it is determined that viewer A has performed a first predetermined action in the first viewer video 1600, the audio of the second broadcaster is played based on the distance (first predetermined distance) between the position where the second avatar 220 of the second broadcaster is displayed in the first viewer video 1600 and the position of viewer A. For example, the volume of the audio of the second broadcaster is determined based on the first predetermined distance in the first viewer video 1600. Specifically, the larger the first predetermined distance, the lower the volume of the audio of the second broadcaster that is played. As a result, the audio of the second broadcaster changes depending on the distance, just like in real space, allowing viewer A to feel immersed and enjoy the MR space.

[0187] Furthermore, if the first predetermined distance is equal to or greater than a predetermined value, the audio of the second broadcaster does not need to be played. If the audio of the second broadcaster is not played, there is no need to acquire the audio data of the second broadcaster, thereby reducing the communication load. Furthermore, regardless of whether the first predetermined distance is equal to or greater than a predetermined value, if the second broadcaster is not displayed in the first viewer video 1600 (for example, if viewer A is facing a different direction from the position of the second broadcaster's second avatar), the audio of the second broadcaster does not need to be played. Furthermore, if the first predetermined distance is equal to or greater than a predetermined value and the second broadcaster is not displayed in the first viewer video 1600, the audio of the second broadcaster does not need to be played. Furthermore, regardless of whether the second broadcaster is displayed in the first viewer video 1600, the audio of the second broadcaster may be played if the first predetermined distance is less than a predetermined value.

[0188] FIG. 25 is a diagram showing an example of a first viewer video 1700 in which a favorites list 520 is displayed. The favorites list 520 may be editable by viewer A. The favorites list 520 may be editable on the home screen of FIG. 14 described above, or may be editable while the video is being viewed. The operation of the favorites list 520 may be the same as the home screen of FIG. 14. In this example, "Mr. U," "Mr. V," and "Mr. W" are registered as favorite broadcasters.

[0189] Favorites list 520 may include icon 530 for transitioning to "edit mode." When viewer A selects icon 530, viewer A can edit favorites list 520. By selecting icon 530, viewer A can select icon 540 for changing each broadcaster, icon 550 for switching whether or not to watch each broadcaster, and icon 560 for transitioning to setting screen 570, which will be described later.

[0190] When each broadcaster's icon 540 is selected, the broadcaster can be changed. In FIG. 25, each broadcaster's icon 540 displays "Mr. U," "Mr. V," and "Mr. W." For example, if "Mr. U" is selected, it is possible to change to a specific broadcaster other than "Mr. U," "Mr. V," and "Mr. W." It is also possible to delete "Mr. U" (in this case, only "Mr. V" and "Mr. W" will be displayed in the favorites list 520).

[0191] When the icon 550 of each broadcaster is selected, it is possible to change whether or not to watch. In FIG. 25, the icon 550 of each broadcaster displays "Watch." For example, in FIG. 25, when the icon 550 of "Mr. V" is selected, the display of the icon 550 of "Mr. V" changes to "Do not watch." In this case, even if it is determined that a predetermined action has been performed, "Mr. V" will not be displayed in the first viewer video, and either "Mr. U" or "Mr. W" will be displayed. In this case, when the icon 550 of "Mr. V" is selected again, the display of the icon 550 of "Mr. V" will change to "Watch." As described above, viewer A can change the broadcaster to watch as needed.

[0192] FIG. 26 is a diagram showing an example of a first viewer video 1800 that is displayed when a broadcaster's icon 560 is selected in FIG. 25. A settings screen 570 is displayed in the first viewer video 1800. A predetermined action of the broadcaster can be set in the settings screen 570. In FIG. 26, the settings screen 570 allows a predetermined action (appearance action) that causes the broadcaster to appear and a predetermined action (gift action) that grants a gift to the broadcaster to be set. In this example, the settings screen 570 is a settings screen for "Mr. U." The settings screen 570 displays an icon 580 for setting the appearance action, an icon 590 for setting the priority of the appearance action, an icon 630 for setting the gift action, and an icon 650 for setting the content of the gift to be granted.

[0193] 26, a appearing action for displaying "Mr. U" can be set by selecting icon 580. In this example, "clap" is set as the appearing action for displaying "Mr. U."

[0194] In FIG. 26, by selecting icon 590, it is possible to set the priority for displaying "Mr. U" when it is determined that an appearance action has been performed. In this example, "clap" is set as the appearance action for displaying "Mr. U," and its priority is "1." If the appearance action of all of "Mr. U," "Mr. V," and "Mr. W" is "clap," the person with the lower priority number may be displayed first. In other words, when it is determined that viewer A has "clapped," "Mr. U," who has the lower priority number, will be displayed preferentially in the first viewer video.

[0195] Note that if the priorities are the same, one of the broadcasters with the same priority may be displayed randomly. For example, consider a case where the same first predetermined action is performed, and the priorities of "Mr. U" and "Mr. V" are "1" and "Mr. W" are "2." In this case, when the first predetermined action is performed for the first time, either "Mr. U" or "Mr. V" is displayed. When the first predetermined action is performed for the second time, either "Mr. U" or "Mr. V," whichever was not displayed when the first predetermined action was performed, is displayed. Then, when the first predetermined action is performed for the third time, "Mr. W" is displayed. As described above, viewer A can appropriately set the display order of the broadcasters to be viewed. Note that if the priorities are the same, the avatars of broadcasters with the same priority may be displayed in the order in which they are displayed at the top of the favorites list 520 in FIG. 25 (for example, when the priorities are the same, the avatars are displayed in the order of "Mr. U," "Mr. V," and "Mr. W").

[0196] 26, a gift action for giving a gift can be set by selecting icon 630. In this example, "finger heart" is set as the action for giving a gift.

[0197] 26, the gift to be given when the gift action is executed can be set by selecting icon 650. In this example, "hearts" are set as the gift to be given.

[0198] Although the above describes an embodiment related to streaming using avatars, this embodiment can also be applied to aspects other than streaming using avatars. For example, during a game (such as an FPS game or an RPG game), a player can call a teammate (a fellow player in the game) by performing a predetermined action (e.g., clapping). In this case, the initial position and posture of the called teammate may be adjusted taking into account the position and size of objects in the virtual space, such as enemy characters and obstacles such as buildings. If the called teammate is also playing the game at the same time, a notice regarding the destination may be displayed (for example, a message such as "I'm moving because Player A is calling me" or "Be careful as there is a building at your destination" may be displayed on the teammate's screen).

[0199] Note that a teammate may be called on the player's screen after a certain period of time has elapsed since a predetermined action was performed. Performing predetermined actions consecutively places a load on the terminal device. Therefore, by calling a teammate on the player's screen after a certain period of time has elapsed since a predetermined action was performed, the load on the terminal device can be reduced. It is also possible to reduce the stress of the called teammate. Note that the embodiment in which a player calls a teammate during a game is not limited to the example in which a teammate is called when a predetermined action is performed, but a teammate may also be called when a predetermined operation is performed on a controller or keyboard.

[0200] The various embodiments described above can be used in combination with each other unless a contradiction or inconvenience arises. Considering the many possible embodiments to which the principles of the invention disclosed herein can be applied, it should be understood that the illustrated embodiments are merely preferred examples and should not be considered to limit the scope of the claimed invention to these preferred examples. Indeed, the scope of the claimed invention is defined by the appended claims. Accordingly, we claim the grant of a patent as our invention for all inventions falling within the scope of the claimed inventions.

[0201] The computer program of the first aspect can "be executed by at least one processor installed in a terminal device, to cause the at least one processor to function so as to operate based on the actions of the user who is the user of the terminal device, display a first video of the real space as seen from the viewpoint of the user, and, when it is determined that the user has performed a first predetermined action in the first video, display a second video of an animation of an avatar object based on the actions of a first user different from the user in combination with the first video."

[0202] The computer program according to the second aspect is the computer program according to the first aspect described above, wherein "the first predetermined action is an action of the user clapping his / her hands."

[0203] The computer program according to the third aspect is, in the first aspect described above, "displaying the second video in the first video at a position a predetermined distance away from the position of the own user when it is determined that the own user has performed the first predetermined action in the first video."

[0204] The computer program according to the fourth aspect is the same as the first aspect described above, in which "the first predetermined action is an action in which the user touches a first object displayed in the first video."

[0205] The computer program according to the fifth aspect is, in the fourth aspect described above, "causes the at least one processor to function so that, when it is determined that the user has performed the first predetermined action in the first video, the second video is displayed at the position of the first target in the first video."

[0206] The computer program according to the sixth aspect is, in the first aspect described above, "causes the at least one processor to function so that, when the second video is displayed in the first video and it is determined that the user himself has performed the first predetermined action in the first video, a third video relating to animation of an avatar object based on the actions of a second user different from the user himself and the first user is displayed in combination with the first video."

[0207] The computer program according to the seventh aspect is the same as in the sixth aspect described above, in that "when the third video is displayed in combination with the first video and the distance between the position where the second video is displayed in the first video and the position where the third video is displayed is less than a predetermined value, the second video is deleted from the first video."

[0208] The computer program according to the eighth aspect is the same as in the first aspect described above, in which "the first user is determined based on a list previously set by the user, a list relating to the user's distribution history, and a list relating to the user's viewing history."

[0209] The computer program according to the 9th aspect is, in the first aspect described above, "causes the at least one processor to function so that, when it is determined that the user has performed the first predetermined action in the first video, the audio of the first user is played based on the distance between the position where the second video is displayed in the first video and the position of the user."

[0210] The computer program according to the 10th aspect is the same as the 9th aspect described above, and causes the at least one processor to function so that "the volume of the audio is determined based on the distance between the position where the second video is displayed in the first video and the position of the user himself."

[0211] The computer program according to the 11th aspect is the same as the 9th aspect described above, but "causes the at least one processor to function so as not to play the audio of the first user when the distance between the position where the second video is displayed and the position of the user is greater than or equal to a predetermined value."

[0212] The computer program according to the 12th aspect is, in the first aspect described above, that "when the second video is being displayed and it is determined that the user has performed a second predetermined action in the first video that is different from the first predetermined action, causes the at least one processor to function so as to display a fourth video relating to an animation of a gift object related to a gift provided by the user to the first user in combination with the first video and the second video, or to display a fifth video relating to an animation combining the first user's avatar object and the gift object in combination with the first video."

[0213] The computer program according to the 13th aspect is, in the first aspect described above, "causes the at least one processor to change the display of the second video when the second video is displayed in combination with the first video, based on a second object located at a position in the first video where the second video is displayed."

[0214] A computer program according to a 14th aspect is any of the first to 13th aspects, wherein "the at least one processor includes a central processing unit (CPU), a microprocessor, and / or a graphics processing unit (GPU)."

[0215] The method according to the 15th aspect can be "a method executed by at least one processor mounted on a terminal device that executes computer-readable instructions, wherein the at least one processor executes the instructions to operate based on the actions of a user who is the user of the terminal device, and to display a first video of the real space as seen from the viewpoint of the user, and when it is determined that the user has performed a predetermined action in the first video, to display a second video of an animation of an avatar object based on the actions of a first user different from the user, in combination with the first video."

[0216] The method according to the 16th aspect can be configured as in the 15th aspect, where "the at least one processor includes a central processing unit (CPU), a microprocessor, and / or a graphics processing unit (GPU)."

[0217] The system according to the 17th aspect can be "a system including a first terminal device and a server device connected to the first terminal device via a communication line, wherein the server device is configured to transmit to the first terminal device first movement data generated based on the movement of a second user different from the first user of the first terminal device, received from a second terminal device of the second user; the first terminal device is configured to receive the first movement data from the server device, acquire second movement data generated based on the movement of the first user, operate based on the second movement data, display a first video of the real space as seen from the viewpoint of the first user, and, when it is determined that the first user has performed a predetermined action in the first video, display a second video of the animation of an avatar object based on the first movement data in combination with the first video."

[0218] The method according to the 18th aspect can be defined as "a method executed by a system including a first terminal device and a server device connected to the first terminal device via a communication line, the method including: the server device transmitting to the first terminal device first movement data generated based on the movement of a second user different from the first user of the first terminal device, the first movement data being received from a second terminal device of the second user; the first terminal device receiving the first movement data from the server device; the first terminal device acquiring second movement data generated based on the movement of the first user; the first terminal device operating based on the second movement data and displaying a first video relating to the real space as seen from the viewpoint of the first user; and, when it is determined that the first user has performed a predetermined action in the first video, the first terminal device displaying a second video relating to the animation of an avatar object based on the first movement data in combination with the first video."

[0219] The server device according to the 19th aspect can be described as "a server device having at least one processor, the at least one processor being configured to send a web page including a computer program to a terminal device, the computer program being executed by the terminal device that has received the web page, to operate based on the actions of a first user of the terminal device, to display a first video relating to the real space as seen from the viewpoint of the first user, and when it is determined that the first user has performed a predetermined action in the first video, to display a second video relating to the animation of an avatar object based on the actions of a second user different from the first user, in combination with the first video."

[0220] The server device according to the 20th aspect can adopt the configuration of the 19th aspect above, in which "the at least one processor includes a central processing unit (CPU), a microprocessor, and / or a graphics processing unit (GPU)."

[0221] The method according to the 21st aspect can be described as "a method executed by at least one processor installed in a server device that executes computer-readable instructions, the method including the at least one processor executing the instructions to send a web page including a computer program to a terminal device, the computer program being executed by the terminal device that has received the web page, causing the terminal device to function in the following manner: operate based on the actions of a first user of the terminal device, display a first video of the real space as seen from the viewpoint of the first user, and, when it is determined that the first user has performed a predetermined action in the first video, display a second video of an animation of an avatar object based on the actions of a second user different from the first user, in combination with the first video."

[0222] The method of the 22nd aspect can adopt the configuration of the 21st aspect above, where "the at least one processor includes a central processing unit (CPU), a microprocessor, and / or a graphics processing unit (GPU)." [Explanation of symbols]

[0223] 1. Communication Systems 20, 20A, 20B, 20C terminal equipment 30, 30A, 30B, 30C Server equipment

Claims

1. When executed by at least one processor installed in the terminal device, a first video that operates based on a motion of a user who is a user of the terminal device and that displays a first video related to a real space that can be seen from a viewpoint of the user; When it is determined that the user has performed a first predetermined action in the first video, a second video relating to an animation of an avatar object based on an action of a first user different from the user is displayed in combination with the first video. a computer program product causing the at least one processor to function in accordance with the instructions of the computer program product;

2. The computer program according to claim 1 , wherein the first predetermined action is a movement based on a hand movement of the user.

3. When it is determined that the user has performed the first predetermined action in the first video, the second video is displayed in the first video at a position that is a predetermined distance away from the position of the user.

2. The computer program product of claim 1, further comprising:

4. The computer program according to claim 1 , wherein the first predetermined action is an action of the user touching a first object displayed in the first video.

5. When it is determined that the user has performed the first predetermined action in the first video, the second video is displayed at a position of the first target in the first video.

5. The computer program product of claim 4, further comprising: a processor configured to:

6. When the second video is displayed in the first video and it is determined that the user has performed the first predetermined action in the first video, a third video relating to an animation of an avatar object based on an action of a second user different from the user and the first user is displayed in combination with the first video.

2. The computer program product of claim 1, further comprising:

7. When the third video is displayed in combination with the first video, and the distance between a position where the second video is displayed and a position where the third video is displayed in the first video is equal to or less than a predetermined value, the second video is deleted from the first video.

7. The computer program product of claim 6, further comprising: a processor configured to:

8. The first user is determined based on the viewing history of the first user.

2. The computer program of claim 1.

9. When it is determined that the user has performed the first predetermined action in the first video, a voice of the first user is played back based on a distance between a position where the second video is displayed in the first video and a position of the user.

2. The computer program product of claim 1, further comprising:

10. the volume of the audio is determined based on the distance between a position where the second video is displayed in the first video and the position of the user.

10. The computer program product of claim 9, further comprising:

11. not playing back the audio of the first user when the distance between the position where the second video is displayed and the position of the user is equal to or greater than a predetermined value; 10. The computer program product of claim 9, further comprising:

12. When the second video is being displayed and it is determined that the user has performed a second predetermined action in the first video that is different from the first predetermined action, a fourth video relating to an animation of a gift object relating to a gift that the user will provide to the first user is displayed in combination with the first video and the second video, or a fifth video relating to an animation combining the avatar object of the first user and the gift object is displayed in combination with the first video.

2. The computer program product of claim 1, further comprising:

13. When the second moving image is displayed in combination with the first moving image, the display of the second moving image is changed based on a second object located at a position in the first moving image where the second moving image is displayed.

2. The computer program product of claim 1, further comprising:

14. The computer program product of claim 1 , wherein the at least one processor comprises a central processing unit (CPU), a microprocessor, and / or a graphics processing unit (GPU).

15. 1. A method performed by at least one processor on a terminal device executing computer-readable instructions, comprising: Execution of the instructions by the at least one processor results in: A first video that operates based on a movement of a user who is a user of the terminal device and that is related to a real space visible from a viewpoint of the user; and When it is determined that the user has performed a predetermined action in the first video, a second video relating to animation of an avatar object based on an action of a first user different from the user is displayed in combination with the first video; A method comprising:

16. The method of claim 15 , wherein the at least one processor comprises a central processing unit (CPU), a microprocessor, and / or a graphics processing unit (GPU).

17. A system including a first terminal device and a server device connected to the first terminal device via a communication line, the server device transmits, to the first terminal device, first action data generated based on an action of a second user different from the first user of the first terminal device and received from a second terminal device of the second user; It is configured as follows: The first terminal device receiving the first motion data from the server device; acquiring second action data generated based on the action of the first user; operating based on the second action data and displaying a first video relating to a real space seen from a viewpoint of the first user; When it is determined that the first user has performed a predetermined action in the first video, a second video relating to animation of an avatar object based on the first action data is displayed in combination with the first video. The system is configured as follows:

18. A method executed by a system including a first terminal device and a server device connected to the first terminal device via a communication line, the server device transmits, to the first terminal device, first action data generated based on an action of a second user different from the first user of the first terminal device and received from the second terminal device of the second user; the first terminal device receiving the first operation data from the server device; The first terminal device acquires second motion data generated based on a motion of the first user; The first terminal device operates based on the second operation data and displays a first video relating to a real space visible from a viewpoint of the first user; and when it is determined that the first user has performed a predetermined action in the first video, the first terminal device displays a second video relating to animation of an avatar object based on the first action data in combination with the first video; A method comprising:

19. A server device comprising at least one processor, the at least one processor is configured to transmit a web page including a computer program to a terminal device; The computer program When the terminal device receives the web page and executes the a first video that operates based on a motion of a first user of the terminal device and that displays a first video relating to a real space seen from a viewpoint of the first user; When it is determined that the first user has performed a predetermined action in the first video, a second video showing animation of an avatar object based on an action of a second user different from the first user is displayed in combination with the first video. A server device that causes the terminal device to function as follows.

20. 20. The server device of claim 19, wherein the at least one processor comprises a central processing unit (CPU), a microprocessor, and / or a graphics processing unit (GPU).

21. 1. A method performed by at least one processor on a server device executing computer-readable instructions, comprising: Execution of the instructions by the at least one processor results in: transmitting a web page including a computer program to a terminal device; The computer program When the web page is received by the terminal device, a first video that operates based on a motion of a first user of the terminal device and that displays a first video relating to a real space seen from a viewpoint of the first user; When it is determined that the first user has performed a predetermined action in the first video, a second video showing animation of an avatar object based on an action of a second user different from the first user is displayed in combination with the first video. The method of causing the terminal device to function in such a manner.

22. 22. The method of claim 21, wherein the at least one processor comprises a central processing unit (CPU), a microprocessor, and / or a graphics processing unit (GPU).

Images