Information processing system, information processing device, distribution method, and distribution program

Abstract

To provide a mechanism in which an image of a person that is acquired from a camera can be flexibly utilized in a distribution destination.SOLUTION: An information processing system comprises: a distribution terminal that distributes first data which includes image data of a game being played, and second data which includes imaged image data acquired from a camera and area data indicating an area of a person included in the imaged image data; and a reception terminal that receives the first data and the second data from the distribution terminal, and generates a distribution display image, in which the area of the person of the imaged image data is displayed by overlapping with an image of the game being played using the distribution terminal, on the basis of the area data.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing apparatus, a distribution method, and a distribution program.

Background Art

[0002] A game system including a camera is known (for example, Japanese Patent Application Laid-Open No. 2016-126042 (Patent Document 1)).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides a mechanism that can flexibly use a person's image acquired from a camera at a distribution destination.

Means for Solving the Problems

[0005] (Configuration 1) An information processing system according to an embodiment includes a distribution terminal that distributes first data including image data of a game in progress and second data including captured image data acquired from a camera and region data indicating a region of a person included in the captured image data, and a reception terminal that receives the first data and the second data from the distribution terminal, and generates a distribution display image in which a region of a person in the captured image data is superimposed and displayed on an image of a game in progress on the distribution terminal based on the region data.

[0006] According to Configuration 1, for example, the receiving terminal can overlay an image of a person captured by the distribution terminal's camera onto the image of the game being played on the distribution terminal, allowing users of the receiving terminal to easily recognize the player of the game distributing the image. Since the first data, which includes image data, and the second data, which includes captured image data and region data, are distributed independently, the receiving terminal can simplify the processing of generating game images based on the image data, and the processing of generating images of people based on the captured image data and region data.

[0007] (Configuration 2) In Configuration 1, in the second data, the captured image data may be compressed using the first compression method, and the region data may be compressed using a second compression method different from the first compression method. According to Configuration 2, by employing compression methods suitable for both the captured image data and the region data, it is possible to generate a person image that does not appear unnatural.

[0008] (Configuration 3) In Configuration 1 or 2, the second compression method may have less data loss compared to the first compression method. Region data is relatively more affected by data loss in terms of quality, while captured image data is relatively less affected by data loss in terms of quality. Therefore, according to Configuration 3, the amount of data in the second data can be efficiently reduced while maintaining the quality of the generated person image.

[0009] (Configuration 4) In Configuration 2 or 3, the first compression method may be lossy compression. The second compression method may be lossless compression. Region data has less data volume compared to captured image data. By adopting lossless compression as the compression method for region data, it is possible to suppress the occurrence of unnaturalness due to errors in the region of a person, etc.

[0010] (Configuration 5) In any of Configurations 1 to 4, the second data may be stream data conforming to the H.264 standard. The second data is imagingThe configuration may include image stream data containing image data and non-image stream data containing region data. According to configuration 5, since the image stream data containing image data and the non-image stream data containing region data are included in the same stream data, the temporal lag between the image data and the region data can be reduced.

[0011] (Configuration 6) In Configuration 5, each image stream data may be a NAL unit containing control information indicating that it contains image data and captured image data for one frame. Each non-image stream data may be a NAL unit containing control information indicating that it contains non-image data and region data for one frame. According to Configuration 6, image data and region data can be distributed using the same stream data, using the structure of the NAL unit defined in the H.264 standard.

[0012] (Configuration 7) In any of Configurations 1 to 6, the receiving terminal is 2 Based on the region data included in the data, the region of the captured image data included in the first data, excluding the region of the person, may be set so that the image of the game currently being played on the superimposed destination is displayed preferentially. According to configuration 7, since the image to be displayed preferentially is determined based on the region data, the image of the person can be generated, thus simplifying the image generation process at the receiving terminal.

[0013] (Configuration 8) In any of Configurations 1 to 7, the region data may be generated based on the outline of a person included in the captured image data. According to Configuration 8, only the person included in the captured image data can be superimposed on the game image.

[0014] (Configuration 9) According to one embodiment, an information processing device is provided that is network-connected to one or more distribution terminals. The distribution terminal distributes first data, which includes image data of a game being played on the distribution terminal, and second data, which includes captured image data acquired from a camera and region data indicating the region of a person included in the captured image data. The information processing device includes a receiving unit that receives the first data and the second data from the distribution terminal, and a generating unit that generates a distributed display image in which the region of a person in the captured image data is superimposed on the image of the game being played on the distribution terminal, based on the region data.

[0015] (Configuration 10) A distribution method according to one embodiment includes the steps of: a distribution terminal distributing first data including image data of a game being played on the distribution terminal and second data including captured image data acquired from a camera and region data indicating the region of a person included in the captured image data; a receiving terminal receiving the first data and the second data from the distribution terminal; and the receiving terminal receiving the first data and the second data and generating a distribution display image in which the region of a person in the captured image data is superimposed on the image of the game being played on the distribution terminal, based on the region data.

[0016] (Configuration 11) A distribution program according to one embodiment delivers to a network-connected computer the following steps: first data including image data of a game running on the computer, second data including captured image data acquired from a camera and region data indicating the region of a person included in the captured image data; and fourth data receiving from another computer the following steps: third data including image data of a game running on the other computer, fourth data including captured image data acquired from a camera and region data indicating the region of a person included in the captured image data, and based on the region data included in the third data, it generates a distribution display image in which the region of a person in the captured image data included in the fourth data is superimposed on the image of the game running on the other computer. [Brief explanation of the drawing]

[0017] [Figure 1] It is a schematic diagram showing a configuration example of a game system according to this embodiment. [Figure 2] It is a schematic diagram showing a hardware configuration example of a game device according to this embodiment. [Figure 3] It is a schematic diagram showing a hardware configuration example of a distribution server according to this embodiment. [Figure 4] It is a schematic diagram showing an example of a screen image displayed on a display of a game device according to this embodiment. [Figure 5] It is a schematic diagram showing an example of a display mode of a camera image displayed on a display of a game device according to this embodiment. [Figure 6] It is a schematic diagram showing an example of a data structure of distribution data in a game system according to this embodiment. [Figure 7] It is a diagram for explaining an example of mask data in a game system according to this embodiment. [Figure 8] It is a schematic diagram showing an example of a data structure of received data in a game system according to this embodiment. [Figure 9] It is a schematic diagram showing an example of a software configuration in a distribution terminal of a game system according to this embodiment. [Figure 10] It is a flowchart showing a processing procedure in a distribution terminal of a game system according to this embodiment. [Figure 11] It is a schematic diagram showing an example of a software configuration in a receiving terminal of a game system according to this embodiment. [Figure 12] It is a flowchart showing a processing procedure in a receiving terminal of a game system according to this embodiment. [Figure 13] It is a flowchart showing a processing procedure in a receiving terminal of a game system according to this embodiment.

Embodiments for Carrying Out the Invention

[0018] This embodiment will be described in detail with reference to the drawings. Note that identical or corresponding parts in the drawings are denoted by the same reference numerals, and their descriptions will not be repeated.

[0019] [A. Example System Configuration] First, an example of the configuration of an information processing system according to this embodiment will be described. Below, an example of the configuration of a game system will be described as an example of an information processing system according to this disclosure.

[0020] Figure 1 is a schematic diagram showing an example configuration of a game system 1 according to this embodiment. The game system 1 includes one or more game devices 100. A game device 100 is an example of an information processing device. Figure 1 shows an example configuration of a game system 1 including four game devices 100. In the following description, when it is necessary to identify each of the multiple game devices 100, they will be distinguished by subnumbers (for example, game devices 100-1, 100-2, 100-3, 100-4 as shown in Figure 1).

[0021] Each game device 100 can distribute image data generated by a game program or the like that executed on its own device to other game devices 100. In addition, each game device 100 can receive image data distributed from other game devices 100 and can display images based on the received image data on its own device's display 120.

[0022] In this specification, the term "distribution" includes the process by which at least one game device 100 (an example of an information processing device) transmits data to one or more other game devices 100. Methods by which one game device 100 distributes data to other game devices 100 include methods of transmitting data via one or more relay entities (e.g., distribution servers 200) and methods of directly transmitting data to a receiving terminal (e.g., P2P). The distributed data may include various types of data in addition to image data.

[0023] The processing related to distribution includes the process by which at least one game device 100 receives data from at least one other game device 100.

[0024] For the sake of explanation, in the following, a game device 100 that distributes image data to other game devices 100 will also be referred to as a "distribution terminal," and a game device 100 that receives image data from other game devices 100 will also be referred to as a "receiving terminal." One game device 100 may serve as both a distribution terminal and a receiving terminal. Furthermore, the game system 1 may also be referred to as a distribution system.

[0025] In this specification, the term "image data" includes moving image data and one or more still image data.

[0026] In the configuration example shown in Figure 1, the game device 100-1 distributes distribution data 10-1, which includes image data generated by the device, to one or more other game devices 100, and also receives received data 20 distributed by one or more other game devices 100. 0 is The display 120 shows an image containing the image to be played based on the received data 20. Game devices 100-2 to 100-4 perform the same processing as game device 100-1.

[0027] In Figure 1, the numbers "1" to "4" in the images displayed on the displays 120 of game devices 100-1 to 100-4 represent the respective images generated by the game programs running on game devices 100-1 to 100-4. As will be explained later, for example, the image represented by the upper "1" and the image represented by the lower "1" displayed on the display 120 of game device 100-1 do not need to be exactly the same; they may have different image quality or be superimposed on each other.

[0028] At least some of the one or more game devices 100 can utilize the camera 118. In the configuration example shown in Figure 1, each of the game devices 100-1 to 100-4 can utilize the camera 118. The distribution data 10 distributed by the game device 100 may include captured image data acquired from the camera 118.

[0029] Game system 1 further includes a distribution server 200 and a management server 300. One or more game devices 100, distribution server 200, and management server 300 are connected to network 4.

[0030] The distribution server 200 relays the data distributed by the game device 100. More specifically, the distribution server 200 receives distribution data 10 from one or more game devices 100 and transmits one or more received data 20, including the one or more received distribution data 10, to one or more game devices 100.

[0031] The management server 300 is responsible for the management necessary for one or more game devices 100 to perform distribution. For example, the management server 300 may decide which one or more game devices 100 will participate in a particular distribution. The management server 300 may decide which user accounts are eligible to participate in the distribution, or it may decide which game devices 100 are eligible to participate in the distribution. In the former case, users associated with the determined user accounts can participate in the distribution. In the latter case, game devices 100 with the determined identification numbers can participate in the distribution.

[0032] The distribution server 200 and the management server 300 may be implemented on a single physical server. The game system 1 may include multiple distribution servers 200 and multiple management servers 300. The distribution server 200 and the management server 300 do not need to have separate functions. The distribution server 200 and the management server 300 may be virtual servers.

[0033] In the following description, the entire group of one or more game devices 100 (or users) participating in a distribution will be referred to as a "distribution group." A distribution group may be dynamically generated in response to a request from any of the game devices 100, or it may be generated in advance. Each game device 100 belonging to a distribution group can receive images from other game devices 100 (i.e., it may be a receiving terminal). In addition, at least some of the game devices 100 belonging to the distribution group can distribute image data to other game devices 100 (i.e., they may be distribution terminals).

[0034] A limit may be set on the number of game devices 100 belonging to the same distribution group (hereinafter also referred to as the "maximum number of participants"). The maximum number of participants may be, for example, 12. A limit may also be set on the number of game devices 100 (distribution terminals) that can distribute image data to other game devices 100 within the same distribution group (hereinafter also referred to as the "maximum number of distributions"). The maximum number of distributions may be, for example, 4.

[0035] [B. Hardware Configuration Example] Next, we will describe an example of the hardware configuration of each device included in the game system 1 according to this embodiment.

[0036] (b1: Game device 100) Figure 2 is a schematic diagram showing an example of the hardware configuration of a game device 100 according to this embodiment. The game device 100 is a type of computer. Referring to Figure 2, the game device 100 includes, for example, one or more processors 102, one or more memories 104, a communication unit 106, a camera interface 108, storage 110, an input unit 116, a display 120, a microphone 122, and a speaker 124.

[0037] The processor 102 is the processing unit for executing processes in the game device 100. The processor 102 includes, for example, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 102 loads the programs stored in the storage 110 into the memory 104 and executes them.

[0038] The memory 104 is a volatile storage medium accessible by the processor 102, and includes, for example, DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory).

[0039] The storage 110 is a non-volatile storage medium accessible by the processor 102, and includes, for example, flash memory or a hard disk. The storage 110 may also include, for example, a storage medium that can be attached to or removed from the game device 100, such as a cartridge or an optical disc.

[0040] The storage 110 stores, for example, a system program 112, a distribution program 114, and a game program 180.

[0041] The system program 112 includes computer-readable instructions for hardware control of the game device 100 and providing a program execution environment.

[0042] The distribution program 114 includes computer-readable instructions for communication processing for distribution and processing necessary for imaging by the camera 118.

[0043] Game program 180 is an example of an application program and includes computer-readable instructions for running a game.

[0044] In this specification, the term "processor" includes, at a minimum, processing circuits that perform processing according to computer-readable instructions, such as CPUs and GPUs, and hardwired circuits, such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field Programmable Gate Arrays).

[0045] In this specification, the term “memory” includes at least the memory 104 and the storage 110.

[0046] In the game device 100, a System on Chip (SoC) in which functions such as a processor and memory are integrated may be used. In this case, the processor and memory reside on the same SoC (integrated circuit). Therefore, in this specification, the terms "processor" and "memory" include both independent configurations and integrated configurations.

[0047] The communication unit 106 sends and receives data to and from other game devices 100, distribution servers 200, management servers 300, etc., via the network 4. The communication unit 106 may be connected to the network 4 by wire or by wireless connection.

[0048] The camera interface 108 transmits and receives data to and from the camera 118. The camera interface 108 may be connected to the camera 118 by a wire or by wireless connection.

[0049] In the communication unit 106 and the camera interface 108, wired connections such as USB (Universal Serial Bus) or parallel connections may be used. For wireless connections, Bluetooth®, ZigBee®, or wireless LAN (IEEE 802.11 standard) may be used.

[0050] The input unit 116 accepts user input. The input unit 116 includes, for example, a keyboard, mouse, game controller, buttons, a directional pad, etc. The input unit 116 may also be an interface with a device that accepts user input (for example, a game controller). That is, the device that accepts user input may be located outside the game device 100.

[0051] The display 120 displays images or videos generated by processing by the processor 102. The display 120 includes, for example, an LCD (Liquid Crystal Display) or an organic EL display. The display 120 may be located outside the game device 100. In this case, the game device 100 may include an interface circuit to the display 120.

[0052] Microphone 122 is located on the casing of the game device 100 and collects sound to generate an audio signal. Speaker 124 is located on the casing of the game device 100 and generates sound from the audio signal. At least one of microphone 122 and speaker 124 may be located outside the game device 100.

[0053] The camera 118 generates camera data (moving image data or still image data) of the subject in accordance with commands from the game device 100 or according to predetermined conditions.

[0054] The camera 118 includes an image sensor, such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. A lens may be placed in front of the image sensor.

[0055] Camera 118 may be, for example, a camera for video calls. Camera 118 may be a dedicated camera that can only be connected to a specific device, or it may be a general-purpose camera that can be connected to any device.

[0056] The camera 118 may be connected to the game device 100 by wire or wireless connection, or it may be incorporated into the game device 100. Multiple cameras 118 may be connected to the game device 100.

[0057] As an example of an information processing device, a game device 100 was given, but the information processing device may also be, for example, a personal computer, television, smartphone, tablet, etc.

[0058] (b2: Distribution server 200) Figure 3 is a schematic diagram showing an example of the hardware configuration of a distribution server 200 according to this embodiment. Referring to Figure 3, the distribution server 200 includes, for example, one or more processors 202, one or more memories 204, one or more communication units 206, storage 210, and an input unit 216.

[0059] Processor 202 is the processing unit for executing processing in the distribution server 200. Processor 202 includes, for example, a CPU and a GPU. Processor 202 loads programs stored in storage 210 into memory 204 and executes them.

[0060] Memory 204 is a volatile storage medium accessible by the processor 202, and includes, for example, DRAM or SRAM.

[0061] The storage 210 is a non-volatile storage medium accessible by the processor 202, and includes, for example, a hard disk or flash memory. The storage 210 stores, for example, a system program 212 and a distribution program 214.

[0062] The system program 212 includes computer-readable instructions for hardware control of the distribution server 200 and for providing a program execution environment.

[0063] The distribution program 214 includes computer-readable instructions for performing relay processing and other operations necessary for distribution, as described later.

[0064] The communication unit 206 sends and receives data to and from the game device 100, the management server 300, etc., via the network 4. The communication unit 206 may be connected to the network 4 by wire or by wireless connection.

[0065] The input unit 216 accepts user input. The input unit 216 includes, for example, a keyboard or mouse.

[0066] (b3: Management Server 300) The hardware configuration example for the management server 300 is the same as the hardware configuration example for the distribution server 200 shown in Figure 3, so a detailed explanation will not be repeated.

[0067] The distribution program stored in the management server 300's storage may include computer-readable instructions for performing the processing necessary for managing distributions, as described later.

[0068] [C. Screen image example] Next, an example of a screen image in the distribution of the game system 1 according to this embodiment will be described.

[0069] Figure 4 is a schematic diagram showing an example of a screen image 126 displayed on the display 120 of a game device 100-1 according to this embodiment. Figure 4 shows an example of a screen image 126 of a game device 100-1 belonging to the distribution group shown in Figure 1. The game device 100-1 operates as at least a receiving terminal.

[0070] Referring to Figure 4, the screen image 126 includes the main image 130 and the distribution display images 140-1 to 140-4 (hereinafter sometimes collectively referred to as "distribution display images 140").

[0071] Each of the distribution display images 140-2 to 140-4 is output based on image data distributed by a distribution terminal belonging to the distribution group. In this embodiment, the game device 100-1 itself is the distribution terminal. Therefore, one distribution display image 140-1 is output based on image data generated in its own device. Each of the other game devices 100-2 to 100-4, Game device 100-1 Based on the distribution data 10 distributed by the device 100, the same distribution display image 140 as distribution display image 140-1 is displayed. Note that the display position of distribution display image 140 may differ depending on each game device 100. If game device 100-1 is not a distribution terminal, all of the distribution display images 140-1 to 140-4 are output based on image data distributed by other distribution terminals.

[0072] Each of the streaming display images 140-1 to 140-4 includes a game image 142, a user icon 144 of the streaming terminal user, a username 146 of the streaming terminal user, and a camera image 148. Note that at least one of the user icon 144 and username 146 may not be displayed. Also, at least one of the game image 142 and camera image 148 may not be displayed. Furthermore, other images may be displayed in addition to or instead of these images.

[0073] The game image 142 is output based on game image data generated by the game program 180 running on the distribution terminal. The user of the distribution terminal may set whether or not to distribute the game image data. The game image 142 is not limited to images generated by the running game program, but may be any image generated on the distribution terminal. For example, the game image 142 may be an image showing a menu screen displayed on the distribution terminal's display 120, or an image generated by the distribution program 114 of the distribution terminal.

[0074] The user icon 144 may be pre-configured or generated by the user of the distribution terminal. The username 146 is displayed based on the user information of the distribution terminal (e.g., the user account name). The image to be displayed as the user icon 144 and the text to be displayed as the username 146 may be received by the game device 100, or generated by the game device 100 based on attribute information of user accounts participating in the distribution group (included in the group information 170 shown in Figure 11) provided by the management server 300.

[0075] The camera image 148 is based on captured image data generated by the camera 118 connected to the distribution terminal. The user of the distribution terminal may be able to set whether or not to distribute the captured image data. For example, if the user of the distribution terminal has not permitted the distribution of captured image data, a predetermined default image (e.g., a black image) may be displayed as the camera image 148. Permission to distribute captured image data on the distribution terminal may be set by the user using the distribution terminal, by another user (e.g., a parent or guardian), or by the distribution terminal or the running application. If the camera 118 is not connected to the distribution terminal, nothing will be displayed in the area where the camera image 148 is displayed, and the background game image 142 may be displayed. The process for displaying the camera image 148 will be described later.

[0076] The main image 130 includes a game image 132. The game image 132 may be output based on game image data (output image data 128, described later) generated by a game program 180 executed in the game device 100-1 (the device itself). In this case, the user of the game device 100-1 can enjoy the game while viewing the game status (game image 142) of other players on the distribution terminal. The game image 132 may also be output based on game image data received from a distribution terminal other than the device itself.

[0077] The resolution (or number of pixels) or frame rate of the game image 132 included in the main image 130 may be higher than that of the game image 142 included in the distribution display image 140.

[0078] [D. Camera Image 148] Next, we will explain the camera image 148 in the distribution display image 140.

[0079] Figure 5 is a schematic diagram showing an example of a display mode for a camera image 148 displayed on the display 120 of a game device 100 according to this embodiment. In Figure 5, the area showing the game image 142 is shown with hatching to clarify the display mode of the camera image 148. In this embodiment, the display mode of the camera image 148 is set on the distribution terminal side, but it may also be set on the receiving terminal side.

[0080] As shown in Figures 5(A) to 5(D), in the streamed display image 140, the camera image 148 is displayed superimposed on the image of the game (game image 142) that is running on the streaming terminal.

[0081] Figure 5(A) shows an example of the contour clipping mode display 148A. The contour clipping mode display 148A includes the portion of the image captured by the distribution terminal that contains a person. In this way, the area of ​​the person in the image captured by the distribution terminal is superimposed on the game image 142 that is running on the distribution terminal. The superimposed area of ​​the person is based on area data. In the camera image superimposed on the game image 142, the game image 142 is displayed in the area other than the area of ​​the person, making it appear as if the area of ​​the person has been cut out and superimposed on the game image 142.

[0082] With the contour clipping mode display 148A, compared to when the game image and the user image are displayed separately, the user of the receiving terminal can easily recognize the person (e.g., the player) who is streaming the game image. Also, the user of the streaming terminal can hide the appearance of the room, etc. Furthermore, by adopting superimposed display, the display area can be used effectively. In addition, it is possible to generate a streaming display image 140 in which the area of ​​the game image 142 that is hidden by the camera image 148 is reduced. In the example shown in Figure 5(A), the area from the chest up of the person is displayed, but for example, the entire area of ​​the person included in the captured image may also be displayed.

[0083] Figure 5(B) shows an example of face mode display 148B. Face mode display 148B includes an image of a predetermined shape that encompasses a person's face, among the images captured by the distribution terminal. This makes it easier for the user of the receiving terminal to check the player's facial expressions during gameplay. The predetermined shape is not limited and may be a circle, ellipse, or polygon, for example. In this way, the region of the predetermined shape that encompasses a person's face among the images captured by the distribution terminal is superimposed on the game image 142 running on the distribution terminal and displayed. In face mode display 148B, the receiving terminal crops the region of the predetermined shape from the captured image data distributed from the distribution terminal. Therefore, the distribution terminal generates and distributes captured image data such that a person's face is located at the position corresponding to the predetermined shape to be cropped.

[0084] Figure 5(C) shows an example of the unprocessed mode display 148C. The unprocessed mode display 148C includes the image captured by the camera 118 of the distribution terminal as is. Therefore, the unprocessed mode display 148C (camera image 148) may include, in addition to the user, objects present in the background of the user as the subject.

[0085] Figure 5(D) shows an example of the background blur mode display 148D. The background blur mode display 148D (camera image 148) includes an image captured by the camera 118 of the distribution terminal in which parts other than people have been blurred.

[0086] [E. Distributed data 10 and received data 20] Next, the distribution data 10 and received data 20 in the distribution of the game system 1 according to this embodiment will be described.

[0087] Figure 6 is a schematic diagram showing an example of the data structure of the distributed data 10 in the game system 1 according to this embodiment.

[0088] Referring to Figure 6, the distribution data 10 may be in a data format suitable for streaming, for example. The distribution data 10 includes, for example, first stream data 11 and second stream data 13.

[0089] The first stream data 11 includes image data necessary for generating the game image 142 of the distribution display image 140. The second stream data 13 includes image data necessary for outputting the camera image 148 of the distribution display image 140.

[0090] The first stream data 11 includes, for example, control information 111 and game image data 113. The control information 111 includes, for example, a bit sequence indicating the start of the stream data, identification information indicating the type of data included in the stream data, and information indicating the data size. The game image data 113 includes image data of the game currently running on the distribution terminal.

[0091] Game image data 113 may include either keyframes (frames that have not undergone inter-frame compression) or difference frames (delta frames).

[0092] The second stream data 13 may include mask stream data 15 and image stream data 14. The mask stream data 15 includes mask data necessary for displaying the contour clipping mode 148A (see Figure 5(A)). The image stream data 14 includes image data necessary for outputting the camera image 148.

[0093] The masked stream data 15 includes, for example, control information 151 and masked data 153. The control information 151 may be the same as the control information 111 of the first stream data 11.

[0094] Mask data 153 is region data that identifies the area of ​​the captured image data 143 that should be displayed as the camera image 148 of the distribution display image 140. Mask data 153 indicates the area of ​​a person included in the captured image data 143.

[0095] The image stream data 14 includes, for example, control information 141 and captured image data 143. The control information 141 may be the same as the control information 111 of the first stream data 11.

[0096] The captured image data 143 is image data acquired from the camera 118 of the distribution terminal. More specifically, the captured image data 143 is generated based on camera data generated by imaging with the camera 118 of the distribution terminal. The camera data output by the camera 118 may be used as is for the captured image data 143, or the camera data output by the camera 118 may be processed before being used for the captured image data 143. Image processing may include, for example, extracting a portion of the camera data, adjusting the image size (reduction / enlargement), and adjusting the image quality (brightness, contrast, color tone, etc.).

[0097] Figure 7 is a diagram illustrating an example of mask data 153 in the game system 1 according to this embodiment.

[0098] Figure 7(A) shows an example where a partial image extracted from the camera data output by camera 118 is used as the captured image data 143. Figure 7(B) shows an example where the entire image of the camera data output by camera 118 is used as the captured image data 143. In the example in Figure 7(B), mask data 153 is shown that was generated so that the upper body of a person is ultimately displayed, but the mask data 153 may also be generated so that, for example, the entire body of a person is ultimately displayed. In the examples shown in Figure 7, all captured image data 143 include objects present in the background as subjects.

[0099] The mask data 153 is generated in association with the captured image data 143. For example, the mask data 153 may be generated based on the outline of a person contained in the captured image data 143. In this case, the area of ​​the person indicated by the mask data 153 can also be said to be the area enclosed by lines that follow the outline of the person.

[0100] The lines following the contour of a person may be positioned slightly outside the contour of the person, and do not necessarily have to strictly follow the contour. For example, the area of ​​the person indicated by the mask data 153 may include an area outside the contour of the person by a predetermined distance. Therefore, the area of ​​the person indicated by the mask data 153 may include the background of the person included in the camera data.

[0101] Any image processing algorithm can be used to identify the contours of a person. The region of the person indicated by the mask data 153 may be determined by any method, not just by being generated based on the contours of the person.

[0102] In the example shown in Figure 7, the mask data 153 includes information for identifying the areas in the captured image data 143 where a person exists and the areas that do not. For example, the mask data 153 is data in which each pixel is assigned a binary value (e.g., "0" and "1") indicating whether or not it is an area of ​​a person. The mask data 153 may also be two-color image data, as in the example shown in Figure 7.

[0103] The receiving terminal uses the mask data 153 and the captured image data 143 included in the second stream data 13 to output a distributed image in which a camera image 148, such as the contour clipping mode display 148A, is displayed. The receiving terminal may process the captured image data 143 so that areas containing people are not effectively displayed. For example, areas where no people exist may be made transparent. The receiving terminal may also generate image data from which the areas containing people have been clipped out of the captured image data 143.

[0104] According to the contour clipping mode display 148A shown in Figure 5(A), the area of ​​the person is identified by the mask data 153, so it is possible to display the area of ​​the person superimposed on the game image 142.

[0105] The example shown involves assigning a binary pixel value to each pixel in the mask data 153, but it is also possible to assign multiple values. For example, a value of "0" could be set to indicate that the display should be disabled for areas where no people exist, or a value of "2" could be set to indicate that the area should be blurred.

[0106] Furthermore, while we explained an example where the area containing a person is assigned the value "1," any value assignment is acceptable as long as it allows you to distinguish between the area containing a person and the other areas.

[0107] The second stream data 13 may include information indicating the display mode corresponding to the captured image data 143. Including information indicating the display mode in the second stream data 13 simplifies or speeds up the generation process of the distributed display image 140 at the receiving terminal. For example, the second stream data 13 may include information indicating that it is a mode in which the outline of a person is cut out and displayed.

[0108] In the second stream data 13, the data compression method for the mask data 153 and the data compression method for the captured image data 143 may be different. For example, since the mask data 153 is binary image data, it may be output in a lossless compression image format such as PNG (Portable Network Graphics) format. Also, the mask data 153 may be compressed using, for example, run-length encoding or Huffman compression, which are examples of lossless compression. In contrast, the captured image data 143 may be compressed using the inter-frame difference method, which is an example of lossy compression. In this case, the captured image data 143 includes either keyframes (frames that have not undergone inter-frame compression) or difference frames. In either case, the data compression method for the mask data 153 results in less data loss compared to the data compression method for the captured image data 143.

[0109] The display mode of the camera image 148 in the distributed display image 140 may be selected by the user of the distribution terminal. If face mode, unprocessed mode, or background blur mode is selected, all pixel values ​​of the mask data 153 may be "1". Alternatively, since the mask data 153 is essentially unnecessary, the second stream data 13 may not include the mask data 153 (or the mask stream data 15 itself).

[0110] The first stream data 11 and the second stream data 13 may be generated according to any video compression standard. For example, the first stream data 11 and the second stream data 13 may be generated according to the H.264 (MPEG-4 AVC) standard. That is, at least a portion of the first stream data 11 and the second stream data 13 may be stream data that conforms to the H.264 standard.

[0111] When generated according to the H.264 standard, each of the mask stream data 15 and image stream data 14 contained in the second stream data 13 corresponds to a Network Abstraction Layer (NAL) unit. That is, the second stream data 13 includes the mask stream data 15 and image stream data 14 separated by NAL units.

[0112] For example, the mask stream data 15 corresponds to a non-VCL-NAL unit that includes mask data 153, which is an example of region data. The control information 151 of the mask stream data 15 may include information indicating that it is a non-VCL-NAL unit. Thus, the mask stream data 15 is a NAL unit that includes control information 151 indicating that the mask stream data 15 contains non-image data, and mask data 153 for one frame.

[0113] The image stream data 14 corresponds to a VCL (Video Coding Layer)-NAL unit, which includes captured image data 143. The control information 141 of the image stream data 14 may include information indicating that it is a VCL-NAL unit, and information indicating a keyframe or difference frame. Thus, the image stream data 14 is a NAL unit that includes control information 141 indicating that the image stream data 14 contains image data, and captured image data 143 for one frame.

[0114] For example, when processing image stream data 14, the mask stream data 15 received immediately beforehand is used. In the second stream data 13, the order in which the mask stream data 15 and the image stream data 14 are arranged does not matter. Also, in the second stream data 13, the image stream data 14 and the mask stream data 15 do not have to be arranged alternately.

[0115] The frame rates of the first stream data 11 and the second stream data 13 may differ from each other. For example, the frame rates of the first stream data 11 and the second stream data 13 may be dynamically changed depending on the communication information, the processing load of the distribution terminal, the number of game devices 100 belonging to the distribution group, the load of the distribution server 200, etc.

[0116] Figure 8 is a schematic diagram showing an example of the data structure of received data 20 in the game system 1 according to this embodiment.

[0117] Referring to Figure 8, the received data 20 includes, for example, distribution data 10 (first stream data 11 and second stream data 13) distributed from each of one or more distribution terminals.

[0118] The frame rates of the first stream data 11 and the second stream data 13 may differ for each distribution terminal.

[0119] If the distribution server 200 relays the distribution data 10 distributed from each of one or more distribution terminals, the distribution server 200 may process part or all of the distribution data 10 before forwarding it to the receiving terminal.

[0120] [F. Example of processing for distribution] Next, we will describe an example of the processing for distribution in the game device 100.

[0121] (f1: streaming device) Figure 9 is a schematic diagram showing an example of the software configuration in a distribution terminal of the game system 1 according to this embodiment. Figure 9 shows an example of the software configuration focusing on the case where the game device 100 operates as a distribution terminal, and for the sake of explanation, other processing modules are not shown.

[0122] Referring to Figure 9, the game device 100, which operates as a distribution terminal, includes a camera data acquisition unit 150, a camera image processing unit 152, a data distribution unit 154, a game program execution unit 156, and a game image processing unit 158.

[0123] For example, the camera data acquisition unit 150, the camera image processing unit 152, the data distribution unit 154, and the game image processing unit 158 ​​are realized by executing the distribution program 114 in an environment where the processor 102 of the game device 100 is executing the system program 112. The game program execution unit 156 is realized by executing the game program 180 in an environment where the processor 102 of the game device 100 is executing the system program 112.

[0124] The camera data acquisition unit 150 acquires camera data 119 output by the camera 118. The camera data acquisition unit 150 may acquire camera data 119 only if the distribution of captured image data is permitted.

[0125] The camera image processing unit 152 generates captured image data 143 based on the camera data 119. The camera image processing unit 152 also generates mask data 153 as needed. As illustrated below, the camera image processing unit 152 performs different generation processes for the captured image data 143 and mask data 153 depending on the display mode.

[0126] When the contour clipping mode is selected, the camera image processing unit 152 identifies the position and contour of the person included as the subject in the camera data 119. Based on the camera data 119, the camera image processing unit 152 generates captured image data 143. The captured image data 143 may be the same as the camera data 119, or it may be image data in which the image size of the camera data 119 has been adjusted based on the position and size of the area containing the person.

[0127] When face mode is selected, the camera image processing unit 152 identifies the position of the person's face included as the subject in the camera data 119. The camera image processing unit 152 extracts a region (e.g., a rectangle) based on the identified position, adjusts the image size of the extracted region, and generates captured image data 143. When face mode is selected, the receiving terminal cuts out a predetermined shape with a set position and size from the received captured image data 143. Therefore, the camera image processing unit 152 determines the region to be extracted from the camera data 119 so that the person's face is located within the predetermined shape region to be cut out by the receiving terminal.

[0128] When the unprocessed mode is selected, the camera image processing unit 152 outputs the camera data 119 as the captured image data 143.

[0129] When the background blur mode is selected, the camera image processing unit 152 identifies the outline of the person included as the subject in the camera data 119. The camera image processing unit 152 blurs the area of ​​the camera data 119 other than the outline of the identified person to generate captured image data 143.

[0130] When contour cropping mode or face mode is selected, the camera image processing unit 152 may output a predetermined default image data (for example, black image data) as the captured image data 143 if it cannot identify a person or a person's face.

[0131] When face mode, no processing mode, or background blur mode is selected, the camera image processing unit 152 may generate mask data 153 in which all pixel values ​​are "1", or it may not generate mask data 153.

[0132] The game program execution unit 156 progresses the game in accordance with user operations, following computer-readable instructions contained in the game program 180. The game program execution unit 156 outputs output image data 128 as image data of the game being played. The output image data 128 has the resolution (or number of pixels) and frame rate that are displayed when the user plays the game on the game device 100.

[0133] The game image processing unit 158 ​​generates game image data 113 from the output image data 128. The game image processing unit 158 ​​generates game image data 113 by, for example, downsampling or downconverting the output image data 128.

[0134] The data distribution unit 154 generates and distributes the first stream data 11 based on the game image data 113.

[0135] The data distribution unit 154 generates and distributes second stream data 13 based on the captured image data 143 and mask data 153.

[0136] Figure 10 is a flowchart showing the processing procedure in the distribution terminal of the game system 1 according to this embodiment. Each step shown in Figure 10 is realized, for example, by the processor 102 of the game device 100 executing the system program 112 and the distribution program 114. It is assumed that the processor 102 of the game device 100 is also executing the game program 180.

[0137] The game device 100 has pre-executed the process for joining a distribution group, and is therefore a member of a certain distribution group and is operating as a distribution terminal within that distribution group.

[0138] Referring to Figure 10, the game device 100 acquires output image data 128 output by the running game program 180 (step S100). The game device 100 generates game image data 113 from the output image data 128 (step S102).

[0139] The game device 100 generates first stream data 11 based on game image data 113 (step S104) and distributes the generated first stream data 11 (step S106).

[0140] The game device 100 determines whether or not it is permitted to distribute the captured image data (step S108).

[0141] If the distribution of captured image data is not permitted (NO in step S108), the game device 100 outputs predetermined image data (for example, black image data in which all pixel values ​​are "0") as captured image data 143 (step S110), and generates mask data 153 in which all pixel values ​​are "0" (step S112).

[0142] If the distribution of captured image data is permitted (YES in step S108), the game device 100 obtains the display mode set by the user (step S114). The game device 100 obtains camera data 119 from the camera 118 (step S116), and processes the camera data 119 according to the display mode to generate captured image data 143 and mask data 153 (step S118). The processing in step S118 is the same as the processing described for the camera image processing unit 152 shown in Figure 9.

[0143] The game device 100 generates second stream data 13 based on the mask data 153 and the captured image data 143 (step S120), and distributes the generated second stream data 13 (step S122).

[0144] The game device 100 determines whether or not it is maintaining a state in which it can operate as a distribution terminal (step S124). If it is maintaining a state in which it can operate as a distribution terminal (YES in step S124), the process from step S100 onward is repeated. If it is not maintaining a state in which it can operate as a distribution terminal (NO in step S124), the process ends.

[0145] Note that mask data 153 does not necessarily have to be generated in step S118. Also, mask data 153 may not be used to generate the second stream data 13 in step S120. Furthermore, the processing in step S112 may be omitted.

[0146] (f2: receiving terminal) Figure 11 is a schematic diagram showing an example of the software configuration in a receiving terminal of the game system 1 according to this embodiment. Figure 11 shows an example of the software configuration focusing on the case where the game device 100 operates as a receiving terminal, and for the sake of explanation, other processing modules are not shown.

[0147] Referring to Figure 11, the game device 100, which operates as a receiving terminal, includes a camera data acquisition unit 150, a camera image processing unit 152, a game program execution unit 156, a game image processing unit 158, a data receiving unit 160, a camera image generation unit 162, a distribution display image generation unit 164, and a screen image generation unit 166.

[0148] For example, the camera data acquisition unit 150, camera image processing unit 152, game image processing unit 158, data reception unit 160, camera image generation unit 162, distribution display image generation unit 164, and screen image generation unit 166 are realized by the distribution program 114 being executed in an environment where the processor 102 of the game device 100 is executing the system program 112. The game program execution unit 156 is realized by the game program 180 being executed in an environment where the processor 102 of the game device 100 is executing the system program 112.

[0149] The camera data acquisition unit 150, camera image processing unit 152, game program execution unit 156, and game image processing unit 158 ​​are activated when the receiving terminal is also a distribution terminal. These processes are the same as those in the distribution terminal shown in Figure 9, so a detailed explanation will not be repeated. However, the camera image processing unit 152 does not generate captured image data 143 and mask data 153, but rather generates camera image data 173 used to display the camera image 148 of the distribution display image 140 for its own device. The camera image data 173 is, for example, image data in which the superimposed image (e.g., game image) is displayed preferentially outside the area of ​​a person. The distribution terminal may generate mask data 153 based on the data used when generating the camera image data 173.

[0150] The game program execution unit 156 outputs output image data 128 as image data of the game being executed.

[0151] The data receiving unit 160 receives the first stream data 11 and the second stream data 13 from each of one or more distribution terminals.

[0152] The camera image generation unit 162 generates camera image data 171 based on the captured image data 143 and mask data 153 included in the second stream data 13. The camera image data 171 is image data in which, for example, the superimposed image (e.g., game image) is displayed preferentially in areas other than the region of a person.

[0153] The camera image generation unit 162 masks the areas of the captured image data 143 other than the area of ​​the person, based on the mask data 153. The masking process includes setting the areas of the captured image data 143 included in the second stream data 13 other than the area of ​​the person, based on the mask data 153 included in the second stream data 13, so that the superimposed game image data 113 (image of the game currently running) is displayed preferentially.

[0154] More specifically, the camera image generation unit 162 may change the pixel values ​​of pixels in the captured image data 143 that correspond to areas other than the area of ​​a person (for example, pixels that are set to "0" in the mask data 153) to values ​​that indicate transparency. Alternatively, the camera image generation unit 162 may decide which pixels from the game image data 113 and the captured image data 143 to use as the pixel values ​​of the corresponding pixels, depending on the value ("0" or "1") indicated by each pixel in the mask data 153.

[0155] The distribution display image generation unit 164 generates distribution display image data 175 based on the game image data 113 and camera image data 171 included in the first stream data 11. More specifically, the distribution display image generation unit 164 superimposes the camera image data 171 onto the game image data 113. The distribution display image generation unit 164 may also generate image data to represent user icons 144 and usernames 146 (see Figure 4) based on group information 170 received from a management server 300 or the like.

[0156] The distribution display image generation unit 164 generates distribution display image data 175 for each distribution terminal.

[0157] The screen image generation unit 166 displays the screen image 126 on the display 120 based on the output image data 128 and the distribution display image data 175 for each distribution terminal. The output image data 128 may be displayed as the main image 130 of the screen image 126.

[0158] In this way, the game device 100, which operates as a receiving terminal, generates a distribution display image 140 that includes images of the game being played on the distribution terminal and captured images, based on the first stream data 11 and the second stream data 13 received from the distribution terminal.

[0159] As described above, in game system 1, the distribution terminal independently distributes game image data 113, which is image data of the game currently running, and image capture image data 143 acquired from the camera 118. The receiving terminal independently receives each data and displays the image of the user of the distribution terminal superimposed on the image of the game currently running on the distribution terminal. By transmitting two types of images independently, the receiving terminal can flexibly utilize the image of a person acquired from the camera 118 of the distribution terminal.

[0160] It is also conceivable that the distribution terminal may superimpose the captured image onto the game image before distribution. However, in this method, for example, if the transmitting terminal is unable to acquire the captured image for some reason, or if the distributor does not want to distribute the captured image, the receiving terminal may not receive the image data itself, including the game image, or the game image may be displayed with an unnaturally cropped portion of the captured image. The game system 1 according to this embodiment suppresses the occurrence of the above-mentioned situations.

[0161] Figures 12 and 13 are flowcharts showing the processing procedure at the receiving terminal of the game system 1 according to this embodiment. Each step shown in Figures 12 and 13 is realized, for example, by the processor 102 of the game device 100 executing the system program 112 and the distribution program 114. It is also assumed that the processor 102 of the game device 100 is executing the game program 180.

[0162] The game device 100 performs the necessary processing in advance to join a distribution group, and by belonging to a certain distribution group, it operates as a receiving terminal within that distribution group.

[0163] Referring to Figures 12 and 13, the game device 100 updates the main image 130 displayed on the display 120 based on the output image data 128 output by the running game program 180 (step S200).

[0164] The game device 100 determines whether or not it is a distribution terminal (step S202). If it is not a distribution terminal (NO in step S202), the processing in steps S204 to S216 is skipped.

[0165] If the device is a distribution terminal (YES in step S202), the game device 100 generates game image data 113 from the output image data 128 (step S204).

[0166] The game device 100 determines whether or not the distribution of captured image data is permitted (step S206).

[0167] If the distribution of captured image data is not permitted (NO in step S206), the game device 100 outputs predetermined image data (for example, default icons) as camera image data 173 (step S208).

[0168] If the distribution of captured image data is permitted (YES in step S206), the game device 100 obtains the display mode set by the user (step S210). The game device 100 obtains camera data 119 from the camera 118 (step S212), and processes the camera data 119 according to the display mode to generate camera image data 173 (step S214). The processing in step S214 is the same as the processing described for the camera image processing unit 152 shown in Figure 9. However, it is not necessary to generate mask data in step S214.

[0169] The game device 100 updates the distribution display image 140 for itself, which is displayed on the display 120, based on the game image data 113 and the camera image data 173 (step S216).

[0170] The game device 100 focuses on one of the distribution terminals (step S218) and receives the first stream data 11 and the second stream data 13 distributed from the focused distribution terminal (step S220). Note that the reception of the first stream data 11 and the reception of the second stream data 13 may be performed independently of each other.

[0171] The game device 100 acquires game image data 113 included in the first stream data 11 from the distribution terminal of interest (step S222). The game device 100 acquires captured image data 143 and mask data 153 included in the second stream data 13 from the distribution terminal of interest (step S224). Note that the second stream data 13 may not include mask data 153. The game device 100 generates camera image data 171 based on the captured image data 143 and mask data 153 (step S226).

[0172] The game device 100 updates the distribution display image 140 for the distribution terminal of interest, which is displayed on the display 120, based on the game image data 113 and the camera image data 171 (step S228).

[0173] The game device 100 determines whether the process of updating the distribution display image 140 has been completed for all distribution terminals (step S230). If there are any distribution terminals for which the process of updating the distribution display image 140 has not been completed (NO in step S230), the process from step S218 onwards is repeated.

[0174] If the process of updating the distribution display image 140 has been completed for all distribution terminals (YES in step S230), the game device 100 determines whether or not its dependency to the distribution group continues (step S232). If the dependency to the distribution group continues (YES in step S232), the process from step S200 onward is repeated. If the dependency to the distribution group has ended (NO in step S232), the process ends.

[0175] [G. Variation] In the above, an example of processing in which mask data 153 is used when contour clipping mode is selected was described. In addition to or instead of contour clipping mode, mask data 153 may also be used in other modes to superimpose the area of ​​a person onto the game image. For example, mask data 153 may be used in face mode. In face mode, an area of ​​a predetermined shape is clipped, so mask data 153 indicating the area to be clipped is distributed. In this modified example, mask data 153 may be information indicating the area of ​​a person, or it may be information indicating the area to be clipped from the distributed captured image data.

[0176] In Figures 11 to 13 above, in a game device 100 that acts as both a distribution terminal and a receiving terminal, the device generates a distribution display image 140 based on camera data output by the device's camera 118 and game image data 113 generated by the game program 180 executed in the device. Alternatively, the device may receive the first stream data 11 and the second stream data 13 that it distributes, and generate the device's distribution display image 140 based on this received data.

[0177] In the above, an example of processing in which captured image data 143 distributed from a distribution terminal is superimposed on game image data 113 (distributed display image 140) distributed from a distribution terminal was described, but the captured image data 143 may be used in any way. For example, the camera image generated based on the captured image data 143 and mask data 153 may be superimposed on any image output by the game program 180 executed on the receiving terminal, or on the main image 130. Furthermore, the camera image may be superimposed on any image output by the receiving terminal (for example, a still image of the background).

[0178] In the above example, if the distribution terminal does not permit the distribution of captured image data, the second stream data 13 is described in which a predetermined default image is included instead of the captured image data 143. In this case, if mask data 153 in which all pixel values ​​are "0" is also distributed at the same time, the camera image 148 will not be displayed on the receiving terminal, regardless of whether the second stream data 13 includes the captured image data 143 or not. Therefore, while the captured image data 143 generated from the camera data is always distributed from the distribution terminal, the display of the camera image 148 on the receiving terminal may be controlled by appropriately changing the pixel values ​​of the mask data 153.

[0179] In the game system 1 according to this embodiment, the distribution terminal may distribute not only images of its own device but also audio spoken by the user. The audio may be distributed together with the images, or it may be distributed as separate stream data.

[0180] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims rather than by the foregoing description, and all modifications within the meaning and scope equivalent to the claims are intended to be included. [Explanation of Symbols]

[0181] 1 Game System, 4 Network, 10 Distribution Data, 11 First Stream Data, 13 Second Stream Data, 14 Image Stream Data, 15 Mask Stream Data, 20 Received Data, 100 Game Device, 102,202 Processor, 104,204 Memory, 106,206 Communication Unit, 108 Camera Interface, 110,210 Storage, 111,141,151 Control Information, 112,212 System Program, 113 Game Image Data, 114,214 Distribution Program, 116,216 Input Unit, 118 Camera, 119 Camera Data, 120 Display, 122 Microphone, 124 Speaker, 126 Screen Image, 128 Output Image Data, 130 Main Image, 132,142 Game Image, 140 Distribution Display Image, 143 Captured Image Data, 144 User Icon, 146 Username, 148 Camera image, 148A Outline cutout mode display, 148B Face mode display, 148C Unprocessed mode display, 148D Background blur mode display, 150 Camera data acquisition unit, 152 Camera image processing unit, 153 Mask data, 154 Data distribution unit, 156 Game program execution unit, 158 Game image processing unit, 160 Data reception unit, 162 Camera image generation unit, 164 Distribution display image generation unit, 166 Screen image generation unit, 170 Group information, 171,173 Camera image data, 175 Distribution display image data, 180 Game program, 200 Distribution server, 300 Management server.

Claims

1. An information processing system, Equipped with a distribution terminal and a receiving terminal, The distribution terminal distributes, respectively, first data including image data of the game currently running on the distribution terminal, and second data including captured image data captured by a camera acquired by the distribution terminal and region data indicating the region of a person included in the captured image data. An information processing system comprising: a receiving terminal which receives the first data and the second data from the distribution terminal and generates a distribution display image in which the region of the person in the captured image data is superimposed on the image of the game being played on the distribution terminal, based on the region data.

2. The information processing system according to claim 1, wherein in the second data, the captured image data is compressed by a first compression method, and the region data is compressed by a second compression method different from the first compression method.

3. The information processing system according to claim 2, wherein the second compression method results in less data loss compared to the first compression method.

4. The first compression method described above is lossy compression, The information processing system according to claim 2, wherein the second compression method is lossless compression.

5. The second data mentioned above is stream data conforming to the H.264 standard, The information processing system according to claim 1, wherein the second data includes image stream data including the captured image data and non-image stream data including the region data.

6. Each of the aforementioned image stream data is a NAL unit that includes control information indicating that image data is included and the captured image data for one frame. The information processing system according to claim 5, wherein each of the non-image stream data is a NAL unit including control information indicating that non-image data is included and region data for one frame.

7. The information processing system according to any one of claims 1 to 6, wherein the receiving terminal sets, based on the region data included in the first data, to display, in a region of the captured image data included in the first data other than the region of the person, with priority given to displaying the image of the game currently being played on which the image is superimposed.

8. The information processing system according to any one of claims 1 to 6, wherein the region data is generated based on the contour of a person included in the captured image data.

9. An information processing device connected to one or more distribution terminals via a network, The distribution terminal distributes, respectively, first data including image data of the game currently running on the distribution terminal, and second data including captured image data captured by a camera acquired by the distribution terminal and region data indicating the region of a person included in the captured image data. The aforementioned information processing device is A receiving unit that receives the first data and the second data from the distribution terminal, An information processing apparatus comprising: a generation unit that generates a distribution display image in which the region of the person in the captured image data is superimposed on the image of the game being played on the distribution terminal, based on the region data.

10. The distribution terminal distributes first data, which includes image data of a game currently running on the distribution terminal, and second data, which includes captured image data taken by a camera acquired by the distribution terminal and region data indicating the region of a person included in the captured image data. The receiving terminal receives the first data and the second data from the distribution terminal, A distribution method comprising the steps of: the receiving terminal receiving the first data and the second data, and generating a distribution display image in which the region of the person in the captured image data is superimposed on the image of the game being played on the distribution terminal, based on the region data.

11. A distribution program that is delivered to network-connected computers. The steps include distributing first data, which includes image data of a game being played on the computer, and second data, which includes captured image data taken by a camera acquired by the computer, and region data indicating the region of a person included in the captured image data, A distribution program that receives from another computer third data including image data of a game running on the other computer, and fourth data including captured image data taken by a camera acquired by the other computer and region data indicating the region of a person included in the captured image data, and performs the step of generating a distribution display image in which the region of the person in the captured image data included in the fourth data is superimposed on the image of the game running on the other computer, based on the region data included in the third data.

Images