System, transmission terminal, reception terminal, method, and program

Abstract

To provide a mechanism capable of realizing an appropriate image display in a device that receives image data generated on the basis of a photographed image of a camera.SOLUTION: There is provided a system comprising at least one transmission terminal and at least one reception terminal. The transmission terminal includes a first image generation unit that generates first image data and a transmission unit that transmits the first image data. The first image generation unit generates, as the first image data, an image of a region that is a part of a photographed image and includes a portion of a human body when the portion of the human body is included in the photographed image of a camera. The transmission unit transmits state data based on the imaging state of the human body in the photographed image. The reception terminal includes a reception unit that receives the first image data and the state data, and a second image generation unit that generates an image for display on a display unit, including the image based on the first image data or the image based on second image data instead of the first image data on the basis of the state data.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to a system, a transmitting terminal, a receiving terminal, a method, and a program. [Background technology]

[0002] A gaming system equipped with a camera is known (for example, Japanese Patent Application Laid-Open No. 2016-126042 (Patent Document 1)). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-126042 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides a mechanism for realizing appropriate image display in a device that receives image data generated based on an image captured by a camera. [Means for solving the problem]

[0005] (Configuration 1) According to one embodiment, there is provided a system including at least one transmitting terminal and at least one receiving terminal. The transmitting terminal includes a first image generation unit that generates first image data and a transmitting unit that transmits the first image data. When a part of a human body is included in an image captured by a camera, the first image generation unit generates an image of a part of the captured image that includes the part of the human body as first image data. The transmitting unit transmits status data based on the captured state of the human body in the captured image. The receiving terminal includes a receiving unit that receives the first image data and the status data, and a second image generation unit that generates an image to be displayed on a display unit based on the status data, the image including an image based on the first image data or an image based on second image data instead of the first image data.

[0006] According to configuration 1, the transmitting terminal transmits the first image data and status data based on the imaging state of the human body in the captured image. The receiving terminal can determine an image to be used to generate an image to be displayed on the display unit based on the status data. This allows the display unit to display an image according to the imaging state of the human body.

[0007] (Configuration 2) In configuration 1, the first image generation unit may generate predetermined image data as the first image data when a first condition is satisfied. The first condition may be a condition that is satisfied when the captured image does not include a part of a human body. The status data may include first data indicating that the captured image does not include a part of a human body. According to configuration 2, when the captured image does not include a part of a human body, the transmitting terminal generates the predetermined image data as the first image data and transmits the first data indicating that the captured image does not include a part of a human body. The receiving terminal can perform necessary processing based on the first data when displaying an image based on the predetermined image data.

[0008] (Configuration 3) In configuration 2, the first image generation unit may generate the predetermined image data as the first image data even when a second condition different from the first condition is satisfied. The transmission unit may transmit the first image data but not transmit the first data when the first condition is not satisfied and the second condition is satisfied. According to configuration 3, the transmitting terminal generates the predetermined image data as the first image data regardless of whether the first condition or the second condition is satisfied. However, if the second condition is satisfied, the second data is not transmitted, and therefore the receiving terminal can display the image differently depending on whether the first condition or the second condition is satisfied.

[0009] (Configuration 4) In configuration 3, the second condition may be a condition that is satisfied when it is selected not to transmit the image captured by the camera. According to configuration 4, the receiving terminal can display an image corresponding to the case where it is selected not to transmit the image captured by the camera.

[0010] (Configuration 5) In any of configurations 1 to 4, when an area including a part of a human body in a captured image is smaller than a predetermined image size, the first image generation unit may generate, as the first image data, an image obtained by enlarging the area including the part of the human body. The status data may include second data indicating the size of the part of the human body included in the captured image. According to configuration 5, the receiving terminal can estimate the quality of the image including the part of the human body based on the second data. The receiving terminal can determine whether or not to display the received captured image depending on the estimated image quality.

[0011] (Configuration 6) In configuration 5, the image size of the first image data may be predetermined. According to configuration 6, the image size exchanged between the transmitting terminal and the receiving terminal is predetermined, which simplifies the image data generation process and the image display process.

[0012] (Configuration 7) In configuration 5 or 6, the first image generation unit may generate the first image data so that the area of ​​the part of the human body in the image of the first image data falls within a range of a predetermined ratio. According to configuration 7, the size of the area in which the part of the human body is displayed can be made uniform, regardless of the positional relationship between the camera and the person.

[0013] (Configuration 8) In any of configurations 5 to 7, the second data may be a magnification factor. According to configuration 8, the receiving terminal can estimate the quality of the image by using the magnification factor calculated when the transmitting terminal generates the captured image.

[0014] (Configuration 9) In any of configurations 1 to 8, the part of the human body may be a person's face. According to configuration 9, the receiving terminal can display an image including the person's face.

[0015] (Configuration 10) In any of configurations 1 to 9, the transmitter may transmit stream data conforming to the H.264 standard, including first image data and status data. The stream data may include a first NAL unit including the first image data and a second NAL unit including the status data. According to configuration 10, the first image data and the status data can be transmitted using the same stream data, using the NAL unit structure defined in the H.264 standard.

[0016] (Configuration 11) According to one embodiment, there is provided a transmitting terminal connected to one or more receiving terminals via a network. The transmitting terminal includes a first image generating unit that generates first image data based on an image captured by a camera, and a transmitting unit that transmits the first image data. When the captured image includes a part of a human body, the first image generating unit generates an image of a part of the captured image that includes the part of the human body as the first image data.

[0017] (Configuration 12) According to one embodiment, there is provided a receiving terminal connected to one or more transmitting terminals via a network. The transmitting terminal includes a first image generation unit that generates first image data and a transmitting unit that transmits the first image data. When a part of a human body is included in an image captured by the camera, the first image generation unit generates an image of a part of the captured image that includes the part of the human body as first image data. The transmitting unit transmits status data based on the captured state of the human body in the captured image. The receiving terminal includes a receiving unit that receives the first image data and the status data, and a second image generation unit that generates an image to be displayed on a display unit based on the status data, the image including an image based on the first image data or an image based on second image data instead of the first image data.

[0018] (Configuration 13) A method according to an embodiment includes a step in which a transmitting terminal generates first image data. When a captured image of a camera includes a part of a human body, an image of a part of the captured image that includes the part of the human body is generated as the first image data. The method includes a step in which the transmitting terminal transmits the first image data and status data based on an imaged state of the human body in the captured image; a step in which the receiving terminal receives the first image data and the status data; and a step in which the receiving terminal generates, based on the status data, an image to be displayed on a display unit, the image including an image based on the first image data or an image based on second image data instead of the first image data.

[0019] (Configuration 14) A program according to an embodiment causes a network-connected computer to execute a step of generating first image data. When a captured image of a camera includes a part of a human body, an image of a part of the captured image that includes the part of the human body is generated as the first image data. The program causes the computer to execute a step of transmitting the first image data and status data based on the captured state of the human body in the captured image, a step of receiving the first image data and the status data, and a step of generating an image to be displayed on a display unit based on the status data, the image including an image based on the first image data or an image based on second image data instead of the first image data. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a schematic diagram showing an example of the configuration of a system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a schematic diagram showing an example of a hardware configuration of a game device according to an embodiment of the present invention. [Figure 3] FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of a distribution server according to the present embodiment. [Figure 4] FIG. 2 is a schematic diagram showing an example of a screen image displayed on a display of the game device according to the present embodiment. [Figure 5]10A and 10B are schematic diagrams showing one example of a display mode of a captured image displayed on a display of the game device according to the present embodiment. [Figure 6] 10A and 10B are diagrams illustrating an example of a generation process of captured image data in the transmitting terminal of the system according to the present embodiment. [Figure 7] FIG. 2 is a schematic diagram showing an example of a data structure of transmission data in the system according to the present embodiment. [Figure 8] FIG. 2 is a schematic diagram showing an example of a software configuration in a transmitting terminal of the system according to the present embodiment. [Figure 9] 10 is a flowchart showing a processing procedure in a transmitting terminal of the system according to the present embodiment. [Figure 10] FIG. 2 is a schematic diagram showing an example of a software configuration in a receiving terminal of the system according to the present embodiment. [Figure 11] 10 is a flowchart showing a processing procedure in a receiving terminal of the system according to the present embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of a user interface screen in a receiving terminal of the system according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0021] The present embodiment will be described in detail with reference to the drawings, in which the same or corresponding parts are designated by the same reference numerals and description thereof will not be repeated.

[0022] [A. System configuration example] First, an example of the configuration of a system according to the present embodiment will be described.

[0023] FIG. 1 is a schematic diagram showing an example configuration of a system 1 according to the present embodiment. System 1 includes one or more game devices 100. Game device 100 is an example of an information processing device. FIG. 1 shows an example configuration of system 1 including four game devices 100. In the following description, when it is necessary to specify each of the multiple game devices 100, they are distinguished by assigning a subnumber (for example, game devices 100-1, 100-2, 100-3, 100-4 shown in FIG. 1).

[0024] Each game device 100 can transmit image data generated by a game program or the like executed on its own device to other game devices 100. Furthermore, each game device 100 can receive image data transmitted from other game devices 100, and can display an image based on the received image data on the display 120 of its own device.

[0025] In this specification, the term "distribution" includes a process in which at least one game device 100 (an example of an information processing device) transmits data to one or more other game devices 100. Methods in which one game device 100 transmits data to another game device 100 include a method in which the data is transmitted via one or more relay entities (e.g., distribution server 200) and a method in which the data is transmitted directly to a receiving terminal (e.g., P2P method, etc.). The transmitted data may include various data in addition to image data.

[0026] The processing related to distribution includes processing in which at least one game device 100 receives data from at least one other game device 100 .

[0027] For ease of explanation, hereinafter, a game device 100 that transmits image data to another game device 100 will also be referred to as a "transmitting terminal," and a game device 100 that receives image data from another game device 100 will also be referred to as a "receiving terminal." One game device 100 may correspond to both a transmitting terminal and a receiving terminal. Therefore, the system 1 includes at least one transmitting terminal and at least one receiving terminal. The system 1 may also be referred to as a distribution system.

[0028] In this specification, the term "image data" includes moving image data and one or more still image data.

[0029] 1, game device 100-1 transmits transmission data 10-1 including image data generated by itself to one or more other game devices 100, and receives transmission data 10-2 to 10-4 transmitted by one or more other game devices 100. Game device 100-1 displays images including images reproduced based on transmission data 10-2 to 10-4 on display 120. Game devices 100-2 to 100-4 execute the same process as game device 100-1.

[0030] 1, the numbers "1" to "4" in the images displayed on display 120 of game devices 100-1 to 100-4 respectively represent images generated by the game programs being executed on game devices 100-1 to 100-4. Note that, as will be described later, for example, among the images displayed on display 120 of game device 100-1, the image represented by the upper "1" and the image represented by the lower "1" do not need to be exactly the same, and they may have different image quality or different images may be superimposed.

[0031] At least some of the one or more game devices 100 can use a camera 118. In the configuration example shown in FIG. 1, each of the game devices 100-1 to 100-4 can use a camera 118. The transmission data 10 transmitted by the game device 100 may include captured image data generated based on an image captured by the camera 118.

[0032] The system 1 further includes a distribution server 200 and a management server 300. One or more game devices 100, the distribution server 200, and the management server 300 are connected to a network 4.

[0033] The distribution server 200 relays data transmitted by the game devices 100. More specifically, the distribution server 200 receives transmission data 10 from one or more game devices 100, and transmits the received transmission data 10 to one or more game devices 100.

[0034] The management server 300 is responsible for the management required for one or more game devices 100 to perform a broadcast. For example, the management server 300 may determine one or more game devices 100 that will participate in a certain broadcast. The management server 300 may determine user accounts that can participate in the broadcast, or may determine identification numbers of game devices 100 that can participate in the broadcast. In the former case, users associated with the determined user accounts can participate in the broadcast. In the latter case, game devices 100 with the determined identification numbers can participate in the broadcast.

[0035] The distribution server 200 and the management server 300 may be implemented on a single physical server. The system 1 may include multiple distribution servers 200 or multiple management servers 300. The distribution server 200 and the management server 300 do not need to have separate functions. The distribution server 200 and the management server 300 may be virtual servers.

[0036] In the following description, one or more game devices 100 (or users) participating in a certain distribution will be collectively referred to as a "distribution group." A distribution group may be dynamically generated in response to a request from one of the game devices 100, or may be generated in advance. Each of the game devices 100 belonging to a certain distribution group can receive images from the other game devices 100 (i.e., they may be receiving terminals). Furthermore, at least some of the game devices 100 belonging to the distribution group can transmit image data to the other game devices 100 (i.e., they may be transmitting terminals).

[0037] An upper limit (hereinafter also referred to as "maximum number of participants") may be set for the number of game devices 100 belonging to the same distribution group. The maximum number of participants may be, for example, 12 devices. An upper limit (hereinafter also referred to as "maximum number of deliveries") may also be set for the number of game devices 100 (transmitting terminals) that can deliver image data to other game devices 100 in the same distribution group. The maximum number of deliveries may be, for example, four devices.

[0038] [B. Hardware configuration example] Next, an example of a hardware configuration of each device included in system 1 according to the present embodiment will be described.

[0039] (b1: game device 100) 2 is a schematic diagram showing an example of a hardware configuration of game device 100 according to the present embodiment. Game device 100 is a type of computer. Referring to FIG. 2, game device 100 includes, for example, one or more processors 102, one or more memories 104, a communication unit 106, a camera interface 108, storage 110, an input unit 116, a display 120, a microphone 122, and a speaker 124.

[0040] Processor 102 is a processing entity for executing processes in game device 100. Processor 102 includes, for example, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). Processor 102 loads a program stored in storage 110 into memory 104 and executes the program.

[0041] The memory 104 is a volatile storage medium accessible by the processor 102, and includes, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0042] Storage 110 is a non-volatile storage medium accessible by processor 102, and includes, for example, a flash memory, a hard disk, etc. Storage 110 may also include, for example, a storage medium that is detachable from game device 100, such as a cartridge or an optical disk.

[0043] The storage 110 stores, for example, a system program 112, a distribution program 114, and a game program 180.

[0044] The system program 112 includes computer-readable instructions for controlling the hardware of the game device 100 and providing a program execution environment.

[0045] The distribution program 114 includes computer-readable instructions for communication processing for distribution, processing required for capturing images with the camera 118, and the like.

[0046] Game program 180 is an example of an application program and includes computer-readable instructions for running a game.

[0047] In this specification, the term "processor" encompasses at least processing circuits that perform processing according to computer-readable instructions, such as a CPU or GPU, and hardwired circuits, such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0048] As used herein, the term “memory” encompasses at least memory 104 and storage 110 .

[0049] Game device 100 may employ an SoC (System on Chip) that integrates functions such as a processor and memory. In this case, the processor and memory exist in the same SoC (Integrated Circuit). Therefore, in this specification, the terms "processor" and "memory" encompass both independent configurations and integrated configurations.

[0050] The communication unit 106 transmits and receives data to and from other game devices 100, the distribution server 200, the management server 300, etc. via the network 4. The communication unit 106 may be connected to the network 4 via a wired connection or may be connected to the network 4 wirelessly.

[0051] The camera interface 108 transmits and receives data to and from the camera 118. The camera interface 108 may be connected to the camera 118 by wire or wirelessly.

[0052] In the communication unit 106 and the camera interface 108, for example, a USB (Universal Serial Bus) connection or a parallel connection may be used as the wired connection, and for example, Bluetooth (registered trademark), ZigBee (registered trademark), wireless LAN (IEEE 802.11 standard), etc. may be used as the wireless connection.

[0053] The input unit 116 accepts user operations. The input unit 116 includes, for example, a keyboard, a mouse, a game controller, buttons, a cross key, etc. The input unit 116 may be an interface with a device that accepts user operations (for example, a game controller). In other words, the device that accepts user operations may be located outside the game device 100.

[0054] Display 120 displays images or videos generated by processing by processor 102. Display 120 includes, for example, an LCD (Liquid Crystal Display) or an organic EL display. Display 120 may be located outside game device 100. In this case, game device 100 may include an interface circuit with display 120.

[0055] The microphone 122 is disposed on the housing of the game device 100 or the like, and collects sound and generates an audio signal. The speaker 124 is disposed on the housing of the game device 100 or the like, and generates sound from the audio signal. At least one of the microphone 122 and the speaker 124 may be located outside the game device 100.

[0056] Camera 118 generates camera data (moving image data or still image data) of an image of a subject in accordance with a command from game device 100 or in accordance with predetermined conditions.

[0057] The camera 118 includes an imaging element such as a charge-coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor, etc. A lens may be disposed in front of the imaging element.

[0058] The camera 118 may be, for example, a camera for video calls. The camera 118 may be a dedicated camera that can be connected only to a specific device, or may be a general-purpose camera that can be connected to any device.

[0059] The camera 118 may be connectable to the game device 100 by wire or wirelessly, or may be built into the game device 100. A plurality of cameras 118 may be connectable to the game device 100.

[0060] Although the game device 100 has been given as an example of an information processing device, the information processing device may be, for example, a personal computer, a television, a smartphone, a tablet, or the like.

[0061] (b2: Distribution server 200) 3 is a schematic diagram showing an example of a hardware configuration of distribution server 200 according to the present embodiment. Referring to FIG. 3, distribution server 200 includes, for example, one or more processors 202, one or more memories 204, one or more communication units 206, storage 210, and input unit 216.

[0062] The processor 202 is a processing entity for executing processes in the distribution server 200. The processor 202 includes, for example, a CPU, a GPU, etc. The processor 202 loads a program stored in the storage 210 into the memory 204 and executes the program.

[0063] The memory 204 is a volatile storage medium accessible by the processor 202, and includes, for example, DRAM, SRAM, and the like.

[0064] The storage 210 is a non-volatile storage medium accessible by the processor 202, and includes, for example, a hard disk, a flash memory, etc. The storage 210 stores, for example, a system program 212 and a distribution program 214.

[0065] The storage 210 is a non-volatile storage medium accessible by the processor 202, and includes, for example, a hard disk, a flash memory, etc. The storage 210 stores, for example, a system program 212 and a distribution program 214.

[0066] The system program 212 includes computer-readable instructions for controlling the hardware of the distribution server 200 and providing a program execution environment.

[0067] The distribution program 214 includes computer-readable instructions for executing relay processing and the like required for distribution, as will be described later.

[0068] The communication unit 206 transmits and receives data to and from the game device 100, the management server 300, and the like via the network 4. The communication unit 206 may be connected to the network 4 via a wired connection or may be connected to the network 4 wirelessly.

[0069] The input unit 216 receives user operations and includes, for example, a keyboard and a mouse.

[0070] (b3: Management Server 300) An example of the hardware configuration of management server 300 is similar to the example of the hardware configuration of distribution server 200 shown in FIG. 3, and therefore detailed description thereof will not be repeated.

[0071] The distribution program stored in the storage of the management server 300 may include computer-readable instructions for executing the processes required for managing distribution as described below.

[0072] [C. Screen image example] Next, an example of a screen image in distribution by system 1 according to the present embodiment will be described.

[0073] Fig. 4 is a schematic diagram showing an example of screen image 126 displayed on display 120 of game device 100-1 according to the present embodiment. Fig. 4 shows an example of screen image 126 of game device 100-1 belonging to the distribution group shown in Fig. 1. Game device 100-1 operates at least as a receiving terminal.

[0074] 4, the screen image 126 includes a main image 130 and delivery display images 140-1 to 140-4 (hereinafter sometimes collectively referred to as "delivery display images 140").

[0075] Each of the distribution display images 140-2 to 140-4 is output based on image data distributed by a transmitting terminal belonging to the distribution group. In this embodiment, the game device 100-1 itself is the transmitting terminal. Therefore, one distribution display image 140-1 is output based on image data generated by the game device itself. Each of the other game devices 100-2 to 100-4 displays the same distribution display image 140 as the distribution display image 140-1 based on the transmission data 10 distributed by the distribution display image 140-1. Note that the display position of the distribution display image 140 may differ depending on each game device 100. If the game device 100-1 is not the transmitting terminal, all of the distribution display images 140-1 to 140-4 are output based on image data distributed by the other transmitting terminals.

[0076] Each of the delivery display images 140-1 to 140-4 includes a game image 142, a user icon 144 of the user of the transmitting terminal, a username 146 of the user of the transmitting terminal, and a camera image 148. At least one of the user icon 144 and the username 146 does not have to be displayed. Also, at least one of the game image 142 and the camera image 148 may not be displayed. Also, another image may be displayed in addition to or instead of these images.

[0077] The game image 142 is output based on game image data generated by the game program 180 running on the transmitting terminal. The user of the transmitting terminal may be able to set whether or not to distribute the game image data. The game image 142 is not limited to an image generated by the running game program, but may be any image generated on the transmitting terminal. For example, the game image 142 may be an image showing a menu screen displayed on the display 120 of the transmitting terminal, or may be an image generated by the distribution program 114 of the transmitting terminal.

[0078] The user icon 144 may be set in advance or generated by the user of the transmitting terminal. The username 146 is displayed based on information of the user of the transmitting terminal (for example, a user account name). The image to be displayed as the user icon 144 and the text to be displayed as the username 146 may be received by the game device 100, or may be generated by the game device 100 based on attribute information of the user accounts participating in the distribution group.

[0079] The camera image 148 is based on captured image data generated by capturing an image with a camera 118 connected to the transmitting terminal. The user of the transmitting terminal may be able to set whether or not to distribute the captured image data. For example, if the user of the transmitting terminal does not permit distribution of captured image data, a predetermined default image (e.g., a black image) may be displayed as the camera image 148. Permission for distribution of captured image data on the transmitting terminal may be set by the user currently using the transmitting terminal, by another user (e.g., a parent), or by the transmitting terminal or an application currently running. Note that if the camera 118 is not connected to the transmitting terminal, nothing may be displayed in the area where the camera image 148 is displayed, and instead, a game image 142 as the background may be displayed. The process for displaying the camera image 148 will be described later.

[0080] The main image 130 includes a game image 132. The game image 132 may be output based on game image data generated by a game program 180 executed on the game device 100 (own device). In this case, the user of the game device 100 can enjoy playing the game himself / herself while viewing the game situation (game image 142) (of other players) on the transmitting terminal.

[0081] Compared to the game image 142 included in the distribution display image 140, the game image 132 included in the main image 130 may have a higher resolution (or number of pixels) or frame rate.

[0082] [D. Camera Image 148] Next, a description will be given of the camera image 148 in the delivery display image 140. The camera image 148 is generated based on the captured image data included in the transmission data .

[0083] Figure 5 is a schematic diagram showing an example of the display mode of camera image 148 displayed on display 120 of game device 100 according to the present embodiment. In Figure 5, the area showing game image 142 is shown hatched to clarify the display mode of camera image 148. In this embodiment, the display mode of camera image 148 is set on the transmitting terminal side, but it may also be set on the receiving terminal side.

[0084] FIG. 5(A) shows an example of face mode display 148A. Face mode display 148A includes an image of a user captured in a predetermined shape with the user's face substantially at the center. This allows the user of the receiving terminal to easily check the facial expressions of players during the game. The predetermined shape may be a circle, ellipse, or polygon. Based on the position of the person's face, an image in which the person's face and the surrounding area are cropped is displayed superimposed on game image 142.

[0085] 5B shows an example of the outline cutout mode display 148B. In the outline cutout mode, the outline cutout mode display 148B includes an image cut out according to the outline of the human body.

[0086] 5(C) shows an example of the no-edit mode display 148C. The no-edit mode display 148C includes the image captured by the camera 118 of the transmitting terminal as is. Therefore, the captured image may include, as a subject, not only the user but also objects present in the background of the user. The image captured by the camera 118 of the transmitting terminal is displayed superimposed on the game image 142.

[0087] 5(D) shows an example of background blur mode display 148D. Background blur mode display 148D includes an image captured by camera 118 of the transmitting terminal in which areas other than the human body are blurred.

[0088] [E. Example of captured image data generation process] Next, an example of the generation process of captured image data used in the face mode display 148A will be described.

[0089] The transmitting terminal processes the camera data output by the camera 118 to generate captured image data. The image size of the captured image data may be determined in advance. When generating the captured image data, the transmitting terminal may identify a part of a human body included in the captured image (camera data) of the camera 118. For example, as shown in FIG. 5(A), the "part of a human body" may be a person's face.

[0090] When face mode display 148A shown in FIG. 5(A) is selected, the transmitting terminal detects the face of a person included in the captured image. The detection of the person's face may be performed using a known face recognition technique. The transmitting terminal extracts partial image data including the detected person's face from the camera data, and generates captured image data 111 from the extracted partial image data.

[0091] FIG. 6 is a diagram illustrating an example of a generation process of captured image data 111 in the transmitting terminal of system 1 according to the present embodiment.

[0092] 6(A) shows an example in which a person's face is included in the field of view of camera 118. The transmitting terminal detects the person's face included in camera data 119 output by camera 118, and determines area 136 including the detected person's face. The transmitting terminal may adjust (reduce / enlarge) the image in determined area 136 so that it matches a preset image size of captured image data 111.

[0093] FIG. 6(B) shows another example in which a person's face is included in the field of view of camera 118. As in FIG. 6(A), the transmitting terminal detects the person's face included in camera data 119 and determines area 136 including the detected person's face. The transmitting terminal enlarges determined area 136 to a predetermined image size to generate captured image data 111. However, in the example shown in FIG. 6(B), the person is located far from camera 118, so area 136 is smaller than the image size of camera data 119. Therefore, area 136 must be enlarged, which may degrade the image quality of the person's face included in captured image data 111.

[0094] In system 1 according to the present embodiment, transmission data transmitted by the transmitting terminal may include information that enables determination of degradation in image quality of a person's face in captured image data 111. This allows the receiving terminal to determine whether or not to generate camera image 148 from the received captured image data 111. For example, if the image quality of a person's face has deteriorated, the receiving terminal may notify the user of the transmitting terminal that camera image 148 cannot be displayed because the user is too far away from camera 118.

[0095] FIG. 6(C) shows an example in which a person's face is not included in the field of view of camera 118. The transmitting terminal cannot detect a person's face in camera data 119. If detection fails, the transmitting terminal may perform any processing, for example, outputting a default image (e.g., a black image) as captured image data 111. In the following description, the default image as shown in FIG. 6(C) is also referred to as a "mute image" in the sense that the captured image is not displayed on the receiving terminal. The mute image is not limited to an image in which all pixels are black, and may be any image. For example, the mute image may be an image in which all pixels are white.

[0096] 6(D) shows an example of processing when distribution of captured images by camera 118 is not permitted. Since distribution is not permitted, the transmitting terminal outputs a default image as captured image data 111. Even in this case, the default image output as captured image data 111 may be a mute image.

[0097] That is, if a person's face cannot be detected from the camera data 119, or if distribution of the captured image data 111 is not permitted, a mute image may be output as the captured image data 111. Therefore, when the receiving terminal receives transmission data 10 including a mute image, it may not be able to determine why the mute image is being transmitted.

[0098] In system 1 according to the present embodiment, the transmission data transmitted by the transmitting terminal may include information indicating whether or not a person's face is included in the field of view of camera 118. That is, the transmission data may include information indicating whether or not a captured image includes a person's face. This allows the receiving terminal to determine that a mute image is being transmitted because no person's face is included in the field of view of camera 118 of the transmitting terminal. In this case, the transmitting terminal may notify the receiving terminal that an image of the user of the transmitting terminal cannot be displayed because no person's face is included in the field of view of camera 118 of the transmitting terminal.

[0099] Note that when outline cutout mode display 148B shown in Fig. 5(B) is selected, if a human face cannot be detected from camera data 119, the area to be cut out cannot be determined, and therefore no valid camera image 148 exists. Therefore, game image 142 may be displayed without being obscured by camera image 148. When no-processing mode display 148C shown in Fig. 5(C) is selected, the transmitting terminal may output camera data 119 as is as captured image data 111. When background blur mode display 148D shown in Fig. 5(D) is selected, the transmitting terminal may generate captured image data 111 by blurring areas other than the human body included in camera data 119.

[0100] [F.Transmission Data 10] Next, the transmission data 10 transmitted by the transmitting terminal will be described.

[0101] FIG. 7 is a schematic diagram showing an example of a data structure of transmission data 10 in system 1 according to the present embodiment.

[0102] 7, the transmission data 10 may be in a data format suitable for streaming, for example. The transmission data 10 includes captured image stream data 11 and game image stream data 12, for example.

[0103] The captured image stream data 11 includes a face presence / absence flag 113 , a distribution permission / denial flag 115 , a magnification ratio 117 , and captured image data 111 .

[0104] The face presence / absence flag 113 is an example of information indicating whether or not a human face is included in the captured image data 111. For example, if the captured image data 111 includes a human face, the face presence / absence flag 113 is set to "1," and if not, the face presence / absence flag 113 is set to "0."

[0105] The distribution permission flag 115 is an example of information indicating whether or not distribution of an image generated by capturing an image with the camera 118 is permitted. For example, if distribution of an image generated by capturing an image with the camera 118 is permitted, the distribution permission flag 115 is set to "1," and if not, the distribution permission flag 115 is set to "0."

[0106] The magnification ratio 117 indicates the ratio of the size of the captured image data 111 to the size of the area 136 set in the camera data 119. Because the image size of the captured image data 111 is predetermined, the magnification ratio 117 is also an index indicating the size of the person's face (part of the human body) included in the captured image. The magnification ratio 117 is an example of information for determining whether the image quality of the person's face included in the captured image data 111 has deteriorated. The larger the magnification ratio 117, the lower the image quality of the person's face in the captured image data 111. In this way, the magnification ratio 117 is information that can be used to determine whether the person's face is captured in a rough state.

[0107] The face presence / absence flag 113 and the magnification ratio 117 may be transmitted only when necessary.

[0108] The captured image data 111 is generated by the image processing described above.

[0109] The game image stream data 12 includes game image data 121 required to generate the game image 142 of the distribution display image 140 .

[0110] The captured image stream data 11 may be generated in accordance with the H.264 (MPEG-4 AVC) standard. The game image stream data 12 may also be generated in accordance with the H.264 standard. That is, the transmitting terminal may transmit stream data in accordance with the H.264 standard, including captured image data 111 and status data based on the imaging status of a human body in the captured image (face presence / absence flag 113, distribution permission flag 115, and magnification ratio 117). Similarly, the transmitting terminal may transmit stream data in accordance with the H.264 standard, including game image data 121.

[0111] Stream data conforming to the H.264 standard is a sequence of data called Network Abstraction Layer (NAL) units. NAL units can be broadly classified into Video Coding Layer (VCL)-NAL units, which contain video information, and non-VCL-NAL units, which do not contain video information.

[0112] When the captured image stream data 11 is generated in accordance with the H.264 standard, for example, one NAL unit (non-VCL-NAL unit) stores status data for the frame (face presence / absence flag 113, distribution permission / prohibition flag 115, and magnification ratio 117) (stream data 11n). One frame of captured image data 111 is stored in one subsequent NAL unit (VCL-NAL unit) (stream data 11v).

[0113] That is, one frame of captured image stream data 11 may be composed of stream data 11v, which is one VCL-NAL unit including captured image data 111, and stream data 11n, which is one non-VCL-NAL unit including status data. On the other hand, one frame of game image stream data 12 may be composed of one VCL-NAL unit including game image data 121. Note that part or all of the status data may be included in the VCL-NAL unit.

[0114] Each of the VCL-NAL units containing captured image data 111 and the VCL-NAL units containing game image data 121 may include either key frames (frames that have not undergone inter-frame compression) or difference frames (delta frames).

[0115] [G. Example of processing for distribution] Next, an example of a process for distribution in the game device 100 will be described.

[0116] (g1: transmitting terminal) Fig. 8 is a schematic diagram showing an example of the software configuration of a transmitting terminal of system 1 according to the present embodiment. Fig. 8 shows an example of the software configuration focusing on the case where game device 100 operates as a transmitting terminal, and for the sake of convenience of explanation, other processing modules are not shown.

[0117] Referring to FIG. 8, game device 100 operating as a transmitting terminal includes a distribution program executing unit 150, a game program executing unit 154, and a transmitting unit 156.

[0118] The distribution program execution unit 150 is realized by the processor 102 of the game device 100 executing the distribution program 114 in an environment in which the system program 112 is being executed. The distribution program execution unit 150 includes a camera image processing unit 151 and a game image processing unit 152.

[0119] The camera image processing unit 151 generates captured image data 111 based on the captured image (camera data 119) of the camera 118. The camera image processing unit 151 generates the captured image data 111 according to whether or not distribution of the captured image data 111 is permitted (distribution permission / denial) and the display mode of the camera image 148.

[0120] As shown in Figure 6(A) above, when a person's face (an example of a part of a human body) is included in the captured image of the camera 118, the camera image processing unit 151 generates an image of a part of the captured image, an area 136 including the part of the human body, as captured image data 111.

[0121] In order to generate captured image data 111 with a predetermined image size, the camera image processing unit 151 may adjust (reduce / enlarge) the image size of partial image data corresponding to the region 136 including a part of the human body. For example, if the region 136 including a part of the human body is smaller than a predetermined image size of the captured image data 111, the camera image processing unit 151 may generate an image of the region 136 stretched as the captured image data 111.

[0122] The position and size of region 136 are determined according to the face of the person to be detected. For example, camera image processing unit 151 generates captured image data 111 so that the area of ​​the person's face (part of the human body) occupying the image of region 136 (captured image data 111) falls within a range of a predetermined ratio. In other words, camera image processing unit 151 determines the position and size of region 136 so that the area other than the face of the person to be detected does not become too large compared to the person's face.

[0123] The camera image processing unit 151 outputs state data based on the imaging state of the human body in the captured image. More specifically, the camera image processing unit 151 outputs face presence / absence information 153, distribution permission / denial information 155, and magnification ratio 157, which are determined in the process of generating the captured image data 111.

[0124] The face presence / absence information 153 is information for determining the value of the distribution permission / denial flag 115. The face presence / absence information 153 indicates whether or not the captured image data 111 includes a human face.

[0125] The distribution permission information 155 is information for determining the value of the face presence / absence flag 113 described above.

[0126] The magnification ratio 157 is information for determining the value of the magnification ratio 117 described above.

[0127] The game program execution unit 154 is realized by the processor 102 of the game device 100 executing the game program 180 in an environment in which the system program 112 is being executed. The game program execution unit 154 progresses the game in response to user operations, in accordance with computer-readable instructions included in the game program 180. The game program execution unit 154 outputs output image data 128 as image data for the game being executed. The output image data 128 has a resolution (or number of pixels) and frame rate that are displayed when the user plays the game on the game device 100.

[0128] The game image processing unit 152 generates game image data 121 from the output image data 128. The game image processing unit 152 generates the game image data 121, for example, by downsampling or downconverting the output image data 128. The game image processing unit 152 may be realized using functions provided by the system program 112, etc.

[0129] The transmitting unit 156 is realized by the processor 102 of the game device 100 executing the system program 112, thereby utilizing the communication unit 106. At least a part of the processing executed by the transmitting unit 156 may be in accordance with the distribution program 114. The transmitting unit 156 generates and transmits captured image stream data 11 based on the captured image data 111, face presence / absence information 153, distribution permission information 155, and magnification ratio 157.

[0130] The transmitting unit 156 generates and transmits game image stream data 12 based on the game image data 121 .

[0131] 9 is a flowchart showing a processing procedure in a transmitting terminal of system 1 according to the present embodiment. Each step shown in FIG. 9 is realized, for example, by processor 102 of game device 100 operating as the transmitting terminal executing system program 112 and distribution program 114. It is assumed that processor 102 of game device 100 also executes game program 180.

[0132] It is assumed that the transmitting terminal has previously executed processing for joining a distribution group, belongs to a certain distribution group, and operates as a transmitting terminal in the distribution group.

[0133] 9, the transmitting terminal acquires output image data 128 output by the currently running game program 180 (step S100). The game device 100 generates game image data 121 from the output image data 128 (step S102).

[0134] The transmitting terminal generates game image stream data 12 based on the game image data 121 (step S104), and transmits the generated game image stream data 12 (step S106).

[0135] The transmitting terminal determines whether or not distribution of the captured image data 111 is permitted (step S108).

[0136] If distribution of the captured image data 111 is not permitted (NO in step S108), the transmitting terminal outputs image data showing a mute image as the captured image data 111 (step S110), and outputs "0" indicating that distribution of the captured image data 111 is not permitted as the distribution permission information 155 (step S112). Note that the transmitting terminal does not need to output the face presence information 153 and the magnification ratio 157.

[0137] The condition that distribution of captured image data 111 is not permitted means a condition that is met when it is selected not to transmit the captured image of camera 118. If distribution of captured image data 111 is not permitted, the transmitting terminal (camera image processing unit 151) outputs a mute image, which is an example of predetermined image data, as captured image data 111. If distribution of captured image data 111 is permitted, the condition described below is not met, and the transmitting terminal does not need to transmit face presence / absence information 153 that indicates that the captured image does not include any part of a human body.

[0138] If distribution of captured image data 111 is permitted (YES in step S108), the transmitting terminal acquires camera data 119 from camera 118 (step S114).

[0139] The transmitting terminal determines the selected display mode (step S116).

[0140] If the face mode is selected ("face mode" in step S116), the transmitting terminal determines whether or not a person's face can be detected in the camera data 119 (step S118).

[0141] If a person's face can be detected (YES in step S118), the transmitting terminal generates captured image data 111 from area 136 including the detected person's face (step S120). The transmitting terminal calculates the magnification ratio when generating captured image data 111 from partial image data corresponding to area 136 (step S122). The transmitting terminal outputs "1" as face presence / absence information 153 indicating that a person's face is included, outputs "1" as distribution permission information 155 indicating that distribution of captured image data 111 is permitted, and outputs magnification ratio 157 when generating captured image data 111 from camera data 119 (step S124).

[0142] If a person's face cannot be detected (NO in step S118), the transmitting terminal outputs a mute image as captured image data 111 (step S126). The condition that a person's face cannot be detected in camera data 119 is an example of a condition that is met when a captured image (camera data 119) does not include any part of a human body. When such a condition is met, the transmitting terminal (camera image processing unit 151) outputs, for example, a mute image as captured image data 111.

[0143] The transmitting terminal outputs "0" as face presence / absence information 153, indicating that the captured image does not contain a human face, outputs "1" as distribution permission information 155, and outputs an invalid value (for example, 0% or a negative value) as magnification ratio 157 (step S128). In this way, the status data includes data indicating that the captured image does not contain any part of a human body. This data can also be said to mean that no part of a human body can be detected in the captured image.

[0144] If any of the no-processing mode, background blur mode, and outline cutout mode is selected ("no-processing mode and others" in step S116), the transmitting terminal generates captured image data 111 from camera data 119 (step S130). The transmitting terminal outputs "1" as distribution permission information 155 (step S132). Here, the transmitting terminal does not need to output face presence / absence information 153 and magnification ratio 157.

[0145] If the face mode is selected, the transmitting terminal may transmit information indicating the outline of the human body in some way. If the background blur mode is selected, the transmitting terminal blurs the area other than the outline of the person in the camera data 119.

[0146] The transmitting terminal generates captured image stream data 11 based on the captured image data 111, face presence / absence information 153, distribution permission information 155, and magnification ratio 157 (step S134), and transmits the generated captured image stream data 11 (step S136). Note that if the face presence / absence information 153 and the magnification ratio 157 have not been output, the transmitting terminal does not need to include the face presence / absence flag 113 and the magnification ratio 117 in the captured image stream data 11.

[0147] The transmitting terminal determines whether or not the state in which it operates as a transmitting terminal is maintained (step S138). If the state in which it operates as a transmitting terminal is maintained (YES in step S138), the processing from step S100 onwards is repeated. If the state in which it operates as a transmitting terminal is not maintained (NO in step S138), the processing ends.

[0148] In addition, in the transmitting terminal, information for supporting the user may be provided according to the result of the process of generating the captured image data 111 from the camera data 119.

[0149] For example, if the face of a person included in camera data 119 cannot be detected (NO in step S118), a message such as "Please move your face so that it is included in the detection range" may be displayed on display 120.

[0150] Furthermore, if the magnification ratio 117 calculated in step S122 exceeds a predetermined threshold value, a message such as "Please move closer to the camera" may be displayed on the display 120 of the transmitting terminal.

[0151] (g2: receiving terminal) Fig. 10 is a schematic diagram showing an example of the software configuration of a receiving terminal of system 1 according to the present embodiment. Fig. 10 shows an example of the software configuration focusing on the case where game device 100 operates as a receiving terminal, and for the sake of convenience of explanation, other processing modules are not shown.

[0152] Referring to FIG. 10, game device 100 operating as a receiving terminal includes a distribution program executing unit 150, a game program executing unit 154, and a receiving unit 160.

[0153] The receiving unit 160 is realized by the processor 102 of the game device 100 executing the system program 112, using the communication unit 106. At least a part of the processing executed by the receiving unit 160 may be in accordance with the distribution program 114. The receiving unit 160 receives captured image stream data 11 and game image stream data 12 from each of one or more transmitting terminals.

[0154] The distribution program execution unit 150 is realized by the processor 102 of the game device 100 executing the distribution program 114 in an environment in which the system program 112 is being executed. The distribution program execution unit 150 includes a captured image generation unit 162, a distribution display image generation unit 164, and a screen image generation unit 166.

[0155] The captured image generation unit 162 generates camera image data 171 based on the captured image data 111 included in the captured image stream data 11. The captured image generation unit 162 determines the generation process for the camera image data 171 based on the face presence / absence flag 113, the distribution permission / prohibition flag 115, and the magnification ratio 117. The captured image generation unit 162 may generate a message 168 based on the face presence / absence flag 113, the distribution permission / prohibition flag 115, and the magnification ratio 117.

[0156] The distribution display image generation unit 164 generates distribution display image data 173 based on the game image data 121 included in the game image stream data 12 and the camera image data 171. The distribution display image generation unit 164 reflects the message 168 in the distribution display image data 173 as necessary (for example, messages 149A, 149B, and 149C shown in FIG. 12 described later).

[0157] The game program execution unit 154 is the same as the game program execution unit 154 shown in FIG.

[0158] The screen image generation unit 166 displays the screen image 126 on the display 120 based on the output image data 128 and the distribution display image data 173. The screen image generation unit 166 updates the display content of the main image 130 based on the output image data 128, and updates the display content of the distribution display images 140-1 to 140-4 (see FIG. 4) based on the distribution display image data 173.

[0159] In addition, when the game device 100 is a transmitting terminal and a receiving terminal, the display content of the distribution display image 140 (for example, the distribution display image 140-1 shown in FIG. 4) corresponding to the device itself may be updated based on the captured image data 111 output by the camera image processing unit 151 (see FIG. 8) and the game image data 121 output by the game image processing unit 152 (see FIG. 8).

[0160] Alternatively, when the game device 100 is a transmitting terminal and a receiving terminal, the display content of the distribution display image 140 corresponding to the device may be updated based on the captured image stream data 11 and game image stream data 12 that the device transmits to other terminals.

[0161] Fig. 11 is a flowchart showing a processing procedure in a receiving terminal of system 1 according to the present embodiment. Each step shown in Fig. 11 is realized, for example, by processor 102 of game device 100 operating as a receiving terminal executing system program 112 and distribution program 114. It is assumed that processor 102 of game device 100 also executes game program 180.

[0162] It is assumed that a receiving terminal performs processing for joining a distribution group in advance, and by belonging to a certain distribution group, operates as a receiving terminal in the distribution group.

[0163] 11, the receiving terminal focuses on one of the transmitting terminals (step S200), and receives the captured image stream data 11 and the game image stream data 12 from the focused transmitting terminal (step S202). Note that the reception of the captured image stream data 11 and the reception of the game image stream data 12 may be performed independently of each other.

[0164] The receiving terminal acquires game image data 121 included in the game image stream data 12 from the transmitting terminal of interest (step S204). The receiving terminal determines whether the value of the distribution permission flag 115 included in the captured image stream data 11 from the transmitting terminal of interest is "0" (i.e., a value indicating that distribution of the captured image data 111 is not permitted) (step S206). If the value of the distribution permission flag 115 is "0" (YES in step S206), the receiving terminal determines camera image data 171 for when distribution is not permitted, and also determines a message 168 indicating that distribution is not permitted (e.g., "Distribution not permitted") (step S208).

[0165] If the value of distribution permission flag 115 is not "0" (NO in step S206), the receiving terminal determines whether the value of face presence / absence flag 113 is "0" (i.e., a value indicating that the captured image does not contain a human face) (step S210). If the value of face presence / absence flag 113 is "0" (YES in step S210), the receiving terminal determines camera image data 171 for when the captured image does not contain a human face, and also determines message 168 indicating that the captured image does not contain a human face (e.g., "No face in the image") (step S212).

[0166] If the value of the face presence / absence flag 113 is not "0" (NO in step S210), the receiving terminal determines whether the value of the magnification ratio 117 exceeds a predetermined threshold value (step S214). If the value of the magnification ratio 117 exceeds the predetermined threshold value (YES in step S214), the receiving terminal determines camera image data 171 for cases where the image quality of the person's face included in the captured image data 111 has deteriorated, and also determines a message 168 (for example, "Too far away") indicating that the image quality of the person's face included in the captured image data 111 has deteriorated (step S216).

[0167] If the value of the magnification ratio 117 does not exceed the predetermined threshold value (NO in step S214), the receiving terminal determines to use the captured image data 111 as camera image data 171 as is (step S218). At this time, the receiving terminal determines that it is not necessary to output the message 168. Note that even if the captured image stream data 11 does not include a valid magnification ratio 117, it may be determined that the value of the magnification ratio 117 does not exceed the predetermined threshold value (NO in step S214).

[0168] The receiving terminal updates the distribution display image 140 for the target transmitting terminal (step S220) based on the game image data 121, the determined camera image data 171, and the message 168. Note that there may be cases where the message 168 does not exist.

[0169] The receiving terminal determines whether the process of updating the delivery display image 140 has been completed for all transmitting terminals (step S222). If there is a transmitting terminal for which the process of updating the delivery display image 140 has not been completed (NO in step S222), the process from step S200 onwards is repeated.

[0170] The receiving terminal updates the main image 130 based on the output image data 128 output by the game program 180 currently being executed on the receiving terminal (step S224).

[0171] The receiving terminal determines whether or not the attachment to the distribution group continues (step S226). If the attachment to the distribution group continues (YES in step S226), the processing from step S200 onward is repeated. If the attachment to the distribution group has ended (NO in step S226), the processing ends.

[0172] In addition, if the receiving terminal is also a transmitting terminal, the receiving terminal may generate the distribution display image 140 of its own terminal based on the camera data 119 output by the camera 118 and the game image generated by the game program 180 running on the receiving terminal, rather than the captured image stream data 11 and game image stream data 12 transmitted by the receiving terminal.

[0173] FIG. 12 is a diagram illustrating an example of a user interface screen in a receiving terminal of system 1 according to the present embodiment.

[0174] Referring to FIG. 12(A), when the receiving terminal determines that the captured image from the transmitting terminal does not contain a human face based on the value of the face presence / absence flag 113 included in the transmission data 10 (after executing step S212), it may display a message 149A such as "No face in the image" in association with the camera image 148.

[0175] Referring to FIG. 12(B), when the receiving terminal determines that the image quality of the person's face included in the captured image data 111 of the transmission data 10 has deteriorated based on the value of the magnification 117 included in the transmission data 10 (after executing step S216), it may display a message 149B such as "Too far away" in association with the camera image 148.

[0176] Referring to FIG. 12(C), when the receiving terminal determines that distribution of the captured image data 111 is not permitted based on the value of the distribution permission flag 115 included in the transmission data 10 (after executing step S208), it may display a message 149C such as "Distribution not permitted" in association with the camera image 148.

[0177] When displaying messages 149A to 149C, a camera image 148 generated based on the captured image data 111 may be displayed. For example, FIG. 12(A) corresponds to a case where the captured image data 111 includes a mute image, and FIG. 12(B) corresponds to a case where the captured image data 111 is an image of low image quality. In FIG. 12(B), the mute image may be displayed as the camera image 148 instead of the captured image data 111. Alternatively, in any of FIGS. 12(A) to 12(C), an arbitrary image generated by the receiving terminal may be displayed as the camera image 148. For example, as shown in FIG. 12(C), an image reflecting information about the user operating the transmitting terminal (e.g., a user account name) may be displayed as the camera image 148. Alternatively, the camera image 148 may be hidden. In this case, the game image 142 is displayed without being obscured by the game image 142.

[0178] Whether or not to display camera image 148 and the type of image to display as camera image 148 may be determined or changed in response to an operation by the user of the receiving terminal.

[0179] In this way, the receiving terminal generates, based on the status data, a distribution display image 140 for display on the display 120, which includes a camera image 148 based on the captured image data 111, or a camera image 148 based on other image data instead of the captured image data 111. The receiving terminal displays a screen image 126 including the generated one or more distribution display images 140.

[0180] Messages 149A to 149C may be displayed all the time while delivery display image 140 is displayed, or may be displayed only while a predetermined condition is satisfied. For example, messages 149A to 149C may be displayed in response to a selection operation of camera image 148 by the user of the receiving terminal.

[0181] Whether or not to display the messages 149A to 149C and the display conditions of the messages 149A to 149C may be determined or changed in response to an operation by the user of the receiving terminal.

[0182] [H. Variations] In the above description, a person's face has been described as an example of a "part of a human body," but the detection target may be not only the person's face but also the whole body. The captured image may include not only a part of the human body (such as the person's face or whole body) but also objects present in the background, but a cut-out image that does not include the background may be output as captured image data (see, for example, FIG. 5(B)).

[0183] In the above description, the captured image stream data 11 (see FIG. 7) includes (the face presence / absence flag 113, the distribution permission / prohibition flag 115, and the magnification ratio 117), but some of these may not be included, or other data may be included instead of or in addition to these as data indicating the image capture state of the human body. For example, if the captured image data 111 includes a human face, the face presence / absence flag 113 may be transmitted, and if the captured image data 111 also includes a human face, the magnification ratio 117 may be transmitted. In this case, the stream data 11n includes only one of the distribution permission / prohibition flag 115 and the magnification ratio 117.

[0184] In the above description, an example was shown in which all of the status data is stored in the stream data 11v (FIG. 7), but at least some of the status data may be stored in different NAL units. For example, the face presence / absence flag 113 may be stored in one NAL unit, and the delivery permission / prohibition flag 115 may be stored in another NAL unit. Storing the face presence / absence flag 113 and the delivery permission / prohibition flag 115 in different NAL units allows for flexible control of the transmission timing of each flag, etc.

[0185] The face presence / absence flag 113 and the magnification ratio 117 may be transmitted only when the face mode is selected. However, even when another mode is selected, at least one of the face presence / absence flag 113 and the magnification ratio 117 may be transmitted.

[0186] At least a part of the status data (face presence / absence flag 113, distribution permission / prohibition flag 115, and magnification ratio 117) included in the captured image stream data 11 may be transmitted by a method other than the captured image stream data 11 (for example, by another stream data). For example, the face presence / absence flag 113 may be transmitted by the captured image stream data 11 (stream data 11n), and the distribution permission / prohibition flag 115 may be transmitted by a different stream. The captured image data 111 needs to be transmitted for each frame, but there are cases where the status data does not need to be transmitted for each frame. Furthermore, there are also cases where at least a part of the status data does not need to be synchronized with the captured image data 111.

[0187] In the above description, face presence / absence information 153 (and face presence / absence flag 113 generated based on face presence / absence information 153) is a binary value indicating whether or not a captured image contains a part of a human body (for example, a person's face), and therefore is data indicating that a captured image does not contain a part of a human body, and also data indicating that a captured image contains a part of a human body. Instead of using such a binary value, information may be added only when a captured image does not contain a part of a human body, or conversely, information may be added only when a captured image contains a part of a human body. In the latter case, the receiving terminal can determine that a captured image does not contain a part of a human body based on the absence of information added.

[0188] Furthermore, instead of the face presence / absence information 153 (and the face presence / absence flag 113 of the captured image stream data 11), the magnification ratio 157 (and the magnification ratio 117 of the captured image stream data 11) may be used as information indicating whether or not a part of a human body (a person's face) is included in the captured image. For example, if the magnification ratio is set to "0" or a value that is not normally used (for example, a negative value), the receiving terminal may determine that the captured image does not include a part of a human body.

[0189] In the above description, the magnification ratio 157 (magnification ratio 117 of the captured image stream data 11) is exemplified as data indicating the size of a person's face included in a captured image. Such data is information that can determine whether the image of the person's face is coarse or not. For example, it can be determined that the larger the magnification ratio 157, the coarser the image (the lower the quality of the image).

[0190] The transmitting terminal may transmit information (a flag) indicating whether or not the magnification ratio 157 exceeds a predetermined threshold value, instead of the magnification ratio 157 itself. Alternatively, the transmitting terminal may transmit the image size of the partial image data corresponding to the region 136, the size of the region of the detected person's face, or the like.

[0191] If a person's face is not present in the captured image, the transmitting terminal may generate and transmit an image containing a message such as "No face in the image" instead of transmitting information indicating that a person's face is not present in the captured image. However, transmitting information indicating that a person's face is not present in the captured image may, for example, increase the degree of processing flexibility in the receiving terminal. For example, the receiving terminal may display a screen image including a mute image, and then display a message or pop-up such as "No face in the image" only after the mute image is selected. Alternatively, the receiving terminal may display information indicating that a person's face is not present in the captured image in a position different from the displayed mute image.

[0192] In the above description, an example of a process for detecting whether or not a part of a human body is included in a captured image when face mode display 148A shown in Fig. 5(A) is selected has been illustrated. In addition to face mode display 148A shown in Fig. 5(A), when outline cutout mode display 148B shown in Fig. 5(B) is selected, it may also be possible to detect whether or not a part of a human body is included in a captured image and transmit the detection result.

[0193] If the transmitting terminal cannot detect the face of a person, the transmitting terminal may notify the user to move the face into the detection range of the camera.

[0194] In the above description, an example of a process has been shown in which the camera image 148 based on the captured image data 111 transmitted by the transmitting terminal is superimposed on a game image, but the camera image 148 based on the captured image data 111 may be used in any way. For example, the camera image 148 may be superimposed on any image output by an application program executed on the receiving terminal. The camera image 148 may also be superimposed on the main image 130. Furthermore, the camera image 148 may be superimposed on any image (for example, a background still image) output by the receiving terminal.

[0195] In system 1 according to the present embodiment, the transmitting terminal may distribute not only the image of the transmitting terminal itself but also the voice of the user. The voice may be distributed together with the image or may be distributed as a separate stream data.

[0196] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0197] 1 system, 4 network, 10 transmission data, 11 captured image stream data, 11n, 11v stream data, 12 game image stream data, 100 game device, 102, 202 processor, 104, 204 memory, 106, 206 communication unit, 108 camera interface, 110, 210 storage, 111 captured image data, 112, 212 system program, 113 face presence / absence flag, 114, 214 distribution program, 115 distribution permission flag, 116, 216 input unit, 117, 157 magnification ratio, 118 camera, 119 camera data, 120 display, 121 game image data, 122 microphone, 124 speaker, 126 screen image, 128 output image data, 130 main image, 132, 142 game image, 136 area, 140 distribution display image, 144 User icon, 146 User name, 148 Camera image, 148A Face mode display, 148B Outline cutout mode display, 148C No processing mode display, 148D Background blur mode display, 149A, 149B, 149C, 168 Message, 150 Distribution program execution unit, 151 Camera image processing unit, 152 Game image processing unit, 153 Face presence information, 154 Game program execution unit, 155 Distribution permission information, 156 Transmission unit, 160 Reception unit, 162 Captured image generation unit, 164 Distribution display image generation unit, 166 Screen image generation unit, 170 Group information, 171 Camera image data, 173 Distribution display image data, 180 Game program, 200 Distribution server, 300 Management server.

Claims

1. A system comprising at least one transmitting terminal and at least one receiving terminal, The transmitting terminal a first image generating unit that generates first image data; a transmission unit that transmits the first image data, when a part of a human body is included in an image captured by a camera, the first image generation unit generates an image of a part of the captured image, the part of the human body being included in the captured image, as the first image data; the transmitting unit transmits status data based on an image capturing status of the human body in the captured image; The receiving terminal a receiving unit that receives the first image data and the status data; A system comprising: a second image generation unit that generates an image to be displayed on a display unit based on the status data, the image including an image based on the first image data or an image based on second image data instead of the first image data.

2. the first image generation unit generates predetermined image data as the first image data when a first condition is satisfied; the first condition is a condition that is satisfied when the captured image does not include a part of a human body, The system of claim 1 , wherein the status data includes first data indicating that the captured image does not include a part of a human body.

3. the first image generation unit generates the predetermined image data as the first image data even when a second condition different from the first condition is satisfied; The system according to claim 2 , wherein the transmitting unit transmits the first image data but does not transmit the first data when the first condition is not satisfied and the second condition is satisfied.

4. The system according to claim 3 , wherein the second condition is a condition that is satisfied when a selection is made not to transmit the image captured by the camera.

5. the first image generation unit generates, as the first image data, an image obtained by enlarging an area including a part of the human body in the captured image when the area includes the part of the human body and the captured image is smaller than a predetermined image size; The system according to claim 1 , wherein the condition data includes second data indicating a size of the part of the human body included in the captured image.

6. The system of claim 5 , wherein the image size of the first image data is predetermined.

7. The system according to claim 5 , wherein the first image generation unit generates the first image data so that an area of ​​the part of the human body occupying the image of the first image data falls within a range of a predetermined ratio.

8. The system of claim 5 , wherein the second data is a magnification ratio.

9. The system according to any one of claims 1 to 8, wherein the part of the human body is a person's face.

10. the transmitting unit transmits stream data conforming to the H.264 standard, including the first image data and the status data; The system according to any one of claims 1 to 8, wherein the stream data includes a first NAL unit including the first image data and a second NAL unit including the status data.

11. A transmitting terminal connected to one or more receiving terminals via a network, a first image generating unit that generates first image data based on an image captured by the camera; a transmission unit that transmits the first image data, The transmitting terminal, wherein, when the captured image includes a part of a human body, the first image generation unit generates an image of a part of the captured image that includes the part of the human body as the first image data.

12. A receiving terminal connected to one or more transmitting terminals via a network, The transmitting terminal a first image generating unit that generates first image data; a transmission unit that transmits the first image data, when a part of a human body is included in an image captured by a camera, the first image generation unit generates an image of a part of the captured image, the part of the human body being included in the captured image, as the first image data; the transmitting unit transmits status data based on an image capturing status of the human body in the captured image; The receiving terminal a receiving unit that receives the first image data and the status data; A receiving terminal comprising: a second image generation unit that generates an image to be displayed on a display unit based on the status data, the image including an image based on the first image data or an image based on second image data instead of the first image data.

13. a transmitting terminal generating first image data, wherein when a part of a human body is included in an image captured by a camera, an image of a part of an area of ​​the captured image, which includes the part of the human body, is generated as the first image data; a step of transmitting, by the transmitting terminal, the first image data and status data based on an imaged state of the human body in the captured image; a receiving terminal receiving the first image data and the status data; and generating, based on the status data, an image for display on a display unit by the receiving terminal, the image including an image based on the first image data or an image based on second image data instead of the first image data.

14. On a networked computer, a step of generating first image data, wherein when a part of a human body is included in an image captured by the camera, an image of a part of the captured image that includes the part of the human body is generated as the first image data; and transmitting the first image data and status data based on an imaged state of the human body in the captured image; receiving the first image data and the status data; and generating an image to be displayed on a display unit, the image including an image based on the first image data or an image based on second image data instead of the first image data, based on the status data.

Images