System, transmitting terminal, receiving terminal, method, and program

Abstract

To provide a mechanism capable of realizing an appropriate image display in a device that receives image data generated on the basis of a photographed image of a camera.SOLUTION: There is provided a system comprising at least one transmission terminal and at least one reception terminal. The transmission terminal includes a first image generation unit that generates first image data and a transmission unit that transmits the first image data. The first image generation unit generates, as the first image data, an image of a region that is a part of a photographed image and includes a portion of a human body when the portion of the human body is included in the photographed image of a camera. The transmission unit transmits state data based on the imaging state of the human body in the photographed image. The reception terminal includes a reception unit that receives the first image data and the state data, and a second image generation unit that generates an image for display on a display unit, including the image based on the first image data or the image based on second image data instead of the first image data on the basis of the state data.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] [[ID=X]] The present disclosure relates to a system, a transmitting terminal, a receiving terminal, a method, and a program.

Background Art

[0002] A game system including a camera is known (for example, Japanese Patent Application Laid-Open No. 2016-126042 (Patent Document 1)).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides a mechanism capable of realizing appropriate image display in an apparatus that receives image data generated based on a captured image of a camera. ;

Means for Solving the Problems

[0005] According to one embodiment, a system including at least one transmitting terminal and at least one receiving terminal is provided. The transmitting terminal includes a first image generation unit that generates first image data, and a transmission unit that transmits the first image data. When a part of a human body is included in the captured image of the camera, the first image generation unit generates, as the first image data, an image of a partial area of the captured image that includes the part of the human body. The transmission unit transmits state data based on the imaging state of the human body in the captured image. The receiving terminal includes a reception unit that receives the first image data and the state data, and a second image generation unit that generates an image for display on a display unit, the image including an image based on the first image data or an image based on second image data instead of the first image data, based on the state data.

[0006] According to Configuration 1, the transmitting terminal transmits first image data and status data based on the imaging state of the human body in the captured image. The receiving terminal can determine the image to be used to generate an image to be displayed on the display unit based on the status data. This allows an image corresponding to the imaging state of the human body to be displayed on the display unit.

[0007] (Configuration 2) In Configuration 1, the first image generation unit may generate predetermined image data as first image data when the first condition is met. The first condition may be a condition that is met when the captured image does not include any part of the human body. The status data may include first data indicating that the captured image does not include any part of the human body. According to Configuration 2, when the captured image does not include any part of the human body, the transmitting terminal generates predetermined image data as first image data and transmits first data indicating that the captured image does not include any part of the human body. When the receiving terminal displays an image based on the predetermined image data, it can perform necessary processing based on the first data.

[0008] (Configuration 3) In Configuration 2, the first image generation unit may generate predetermined image data as the first image data even if a second condition different from the first condition is met. The transmission unit may transmit the first image data but not the first data if the first condition is not met and the second condition is met. According to Configuration 3, the transmission terminal generates predetermined image data as the first image data regardless of whether the first or second condition is met. However, if the second condition is met, the second data is not transmitted, so the receiving terminal can display the image differently depending on whether the first or second condition is met.

[0009] (Configuration 4) In Configuration 3, the second condition may be a condition that is met when it is selected not to transmit the camera's captured image. According to Configuration 4, the receiving terminal can display an image corresponding to the case when it is selected not to transmit the camera's captured image.

[0010] (Configuration 5) In any of Configurations 1 to 4, the first image generation unit may generate an enlarged image of the region containing the human body in the captured image as first image data if the region containing the human body in the captured image is smaller than a predetermined image size. The status data may also include second data indicating the size of the human body region included in the captured image. According to Configuration 5, the receiving terminal can estimate the quality of the image containing the human body region based on the second data. The receiving terminal can determine whether or not to display the received captured image depending on the estimated image quality.

[0011] (Configuration 6) In Configuration 5, the image size of the first image data may be predetermined. According to Configuration 6, since the image size exchanged between the transmitting terminal and the receiving terminal is predetermined, the image data generation process and the image display process can be simplified.

[0012] (Configuration 7) In Configuration 5 or 6, the first image generation unit may generate the first image data such that the area of ​​a part of the human body in the first image data falls within a predetermined proportion. According to Configuration 7, the size of the area in which a part of the human body is displayed can be made uniform, regardless of the positional relationship between the camera and the person.

[0013] (Configuration 8) In any of Configurations 5 to 7, the second data may be the magnification factor. According to Configuration 8, the receiving terminal can estimate the image quality using the magnification factor calculated when the transmitting terminal generates the image captured by the camera.

[0014] (Configuration 9) In any of Configurations 1 to 8, the part of the human body may be a person's face. According to Configuration 9, the receiving terminal can display an image that includes a person's face.

[0015] (Configuration 10) In any of Configurations 1 to 9, the transmitting unit may transmit stream data conforming to the H.264 standard, which includes first image data and status data. The stream data may include a first NAL unit containing first image data and a second NAL unit containing status data. According to Configuration 10, the first image data and status data can be transmitted using the same stream data and the structure of the NAL unit defined in the H.264 standard.

[0016] (Configuration 11) According to one embodiment, a transmitting terminal is provided that is network-connected to one or more receiving terminals. The transmitting terminal includes a first image generation unit that generates first image data based on an image captured by a camera, and a transmitting unit that transmits the first image data. When the captured image includes a part of a human body, the first image generation unit generates an image of a part of the captured image that includes the part of the human body as the first image data.

[0017] (Configuration 12) According to one embodiment, a receiving terminal is provided that is network-connected to one or more transmitting terminals. The transmitting terminal includes a first image generation unit that generates first image data and a transmitting unit that transmits the first image data. The first image generation unit generates an image of a part of the captured image that includes the part of the human body when the image captured by the camera includes a part of the human body. The transmitting unit transmits state data based on the image state of the human body in the captured image. The receiving terminal includes a receiving unit that receives the first image data and state data, and a second image generation unit that generates an image for display on a display unit, based on the state data, which includes an image based on the first image data, or an image based on the second image data instead of the first image data.

[0018] (Configuration 13) A method according to one embodiment includes the step of a transmitting terminal generating first image data. When the image captured by the camera includes a part of a human body, an image of a part of the image that includes the part of the human body is generated as the first image data. The method includes the steps of a transmitting terminal transmitting the first image data and state data based on the imaging state of the human body in the image; a receiving terminal receiving the first image data and state data; and the receiving terminal generating an image for display on a display unit, which includes an image based on the first image data, or an image based on second image data instead of the first image data, based on the state data.

[0019] (Configuration 14) A program according to one embodiment causes a network-connected computer to perform the step of generating first image data. When the image captured by the camera includes a part of the human body, an image of a part of the captured image that includes the part of the human body is generated as the first image data. The program causes the computer to perform the steps of transmitting the first image data and state data based on the imaging state of the human body in the captured image; receiving the first image data and state data; and generating an image for display on a display unit, which includes an image based on the first image data, or an image based on the second image data instead of the first image data, based on the state data. [Brief explanation of the drawing]

[0020] [Figure 1] This is a schematic diagram showing an example of the system configuration according to this embodiment. [Figure 2] This is a schematic diagram showing an example of the hardware configuration of a game device according to this embodiment. [Figure 3] This is a schematic diagram showing an example of the hardware configuration of a distribution server according to this embodiment. [Figure 4] This is a schematic diagram showing an example of a screen image displayed on the display of a game device according to this embodiment. [Figure 5]It is a schematic diagram showing an example of a display mode of a captured image displayed on a display of a game device according to the present embodiment. [Figure 6] It is a diagram for explaining an example of a generation process of captured image data in a transmission terminal of a system according to the present embodiment. [Figure 7] It is a schematic diagram showing an example of a data structure of transmission data in a system according to the present embodiment. [Figure 8] It is a schematic diagram showing an example of a software configuration in a transmission terminal of a system according to the present embodiment. [Figure 9] It is a flowchart showing a processing procedure in a transmission terminal of a system according to the present embodiment. [Figure 10] It is a schematic diagram showing an example of a software configuration in a reception terminal of a system according to the present embodiment. [Figure 11] It is a flowchart showing a processing procedure in a reception terminal of a system according to the present embodiment. [Figure 12] It is a diagram for explaining an example of a user interface screen in a reception terminal of a system according to the present embodiment. [Embodiments for Carrying Out the Invention]

[0021] The present embodiment will be described in detail with reference to the drawings. For the same or corresponding parts in the drawings, the same reference numerals are given and the description thereof will not be repeated.

[0022] [A. System Configuration Example] First, a configuration example of a system according to the present embodiment will be described.

[0023] Figure 1 is a schematic diagram showing an example configuration of System 1 according to this embodiment. System 1 includes one or more game devices 100. Game device 100 is an example of an information processing device. Figure 1 shows an example configuration of System 1 including four game devices 100. In the following description, when it is necessary to identify each of the multiple game devices 100, they will be distinguished by subnumbers (for example, game devices 100-1, 100-2, 100-3, 100-4 shown in Figure 1).

[0024] Each game device 100 can transmit image data generated by a game program or the like that executed on its own device to other game devices 100. In addition, each game device 100 can receive image data transmitted from other game devices 100 and can display an image based on the received image data on its own device's display 120.

[0025] In this specification, the term "distribution" includes the process by which at least one game device 100 (an example of an information processing device) transmits data to one or more other game devices 100. The methods by which one game device 100 transmits data to other game devices 100 include transmitting data via one or more relay entities (e.g., distribution servers 200) and transmitting data directly to a receiving terminal (e.g., P2P). The transmitted data may include various types of data in addition to image data.

[0026] The processing related to distribution includes the process by which at least one game device 100 receives data from at least one other game device 100.

[0027] For the sake of explanation, in the following, a game device 100 that transmits image data to another game device 100 will also be referred to as a "transmitting terminal," and a game device 100 that receives image data from another game device 100 will also be referred to as a "receiving terminal." One game device 100 may also serve as both a transmitting terminal and a receiving terminal. Therefore, System 1 includes at least one transmitting terminal and at least one receiving terminal. System 1 may also be referred to as a distribution system.

[0028] In this specification, the term "image data" includes moving image data and one or more still image data.

[0029] In the configuration example shown in Figure 1, game device 100-1 transmits transmission data 10-1, which includes image data generated by itself, to one or more other game devices 100, and receives transmission data 10-2 to 10-4 transmitted by one or more other game devices 100. Game device 100-1 displays an image on the display 120 that includes an image to be reproduced based on the transmission data 10-2 to 10-4. Game devices 100-2 to 100-4 perform the same processing as game device 100-1.

[0030] In Figure 1, the numbers "1" to "4" in the images displayed on the displays 120 of game devices 100-1 to 100-4 represent the respective images generated by the game programs running on game devices 100-1 to 100-4. As will be explained later, for example, the image represented by the upper "1" and the image represented by the lower "1" displayed on the display 120 of game device 100-1 do not need to be exactly the same; they may have different image quality or be superimposed on each other.

[0031] At least some of the one or more game devices 100 can utilize the camera 118. In the configuration example shown in Figure 1, each of the game devices 100-1 to 100-4 can utilize the camera 118. The transmission data 10 transmitted by the game device 100 may include captured image data generated based on the image captured by the camera 118.

[0032] System 1 further includes a distribution server 200 and a management server 300. One or more game devices 100, the distribution server 200, and the management server 300 are connected to network 4.

[0033] The distribution server 200 relays the data transmitted by the game device 100. More specifically, the distribution server 200 receives transmission data 10 from one or more game devices 100 and transmits the received one or more transmission data 10 to one or more game devices 100.

[0034] The management server 300 is responsible for the management necessary for one or more game devices 100 to perform distribution. For example, the management server 300 may decide which one or more game devices 100 will participate in a particular distribution. The management server 300 may decide which user accounts are eligible to participate in the distribution, or it may decide which game devices 100 are eligible to participate in the distribution. In the former case, users associated with the determined user accounts can participate in the distribution. In the latter case, game devices 100 with the determined identification numbers can participate in the distribution.

[0035] The distribution server 200 and the management server 300 may be implemented on a single physical server. System 1 may include multiple distribution servers 200 or multiple management servers 300. The distribution server 200 and the management server 300 do not need to have separate functions. The distribution server 200 and the management server 300 may be virtual servers.

[0036] In the following description, the entire group of one or more game devices 100 (or users) participating in a distribution will be referred to as a "distribution group." A distribution group may be dynamically generated in response to a request from any of the game devices 100, or it may be generated in advance. Each game device 100 belonging to a distribution group can receive images from other game devices 100 (i.e., it may be a receiving terminal). In addition, at least some of the game devices 100 belonging to the distribution group can transmit image data to other game devices 100 (i.e., they may be transmitting terminals).

[0037] A limit may be set on the number of game devices 100 belonging to the same distribution group (hereinafter also referred to as the "maximum number of participants"). The maximum number of participants may be, for example, 12. A limit may also be set on the number of game devices 100 (transmitting terminals) that can distribute image data to other game devices 100 within the same distribution group (hereinafter also referred to as the "maximum number of distributions"). The maximum number of distributions may be, for example, 4.

[0038] [B. Hardware Configuration Example] Next, we will describe an example of the hardware configuration of each device included in System 1 according to this embodiment.

[0039] (b1: Game device 100) Figure 2 is a schematic diagram showing an example of the hardware configuration of a game device 100 according to this embodiment. The game device 100 is a type of computer. Referring to Figure 2, the game device 100 includes, for example, one or more processors 102, one or more memories 104, a communication unit 106, a camera interface 108, storage 110, an input unit 116, a display 120, a microphone 122, and a speaker 124.

[0040] The processor 102 is the processing unit for executing processes in the game device 100. The processor 102 includes, for example, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 102 loads programs stored in the storage 110 into the memory 104 and executes them.

[0041] The memory 104 is a volatile storage medium accessible by the processor 102, and includes, for example, DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory).

[0042] The storage 110 is a non-volatile storage medium accessible by the processor 102, and includes, for example, flash memory or a hard disk. The storage 110 may also include, for example, a storage medium that can be attached to or removed from the game device 100, such as a cartridge or an optical disc.

[0043] The storage 110 stores, for example, the system program 112, the distribution program 114, and the game program 180.

[0044] The system program 112 includes computer-readable instructions for hardware control of the game device 100 and providing a program execution environment.

[0045] The distribution program 114 includes computer-readable instructions for communication processing for distribution and processing necessary for imaging by the camera 118.

[0046] Game program 180 is an example of an application program and includes computer-readable instructions for running a game.

[0047] In this specification, the term "processor" includes, at a minimum, processing circuits that perform processing according to computer-readable instructions, such as CPUs and GPUs, and hardwired circuits, such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field Programmable Gate Arrays).

[0048] In this specification, the term “memory” includes at least the memory 104 and the storage 110.

[0049] In the game device 100, a System on Chip (SoC) in which functions such as a processor and memory are integrated may be used. In this case, the processor and memory reside on the same SoC (integrated circuit). Therefore, in this specification, the terms "processor" and "memory" include both independent configurations and integrated configurations.

[0050] The communication unit 106 sends and receives data to and from other game devices 100, distribution servers 200, management servers 300, etc., via the network 4. The communication unit 106 may be connected to the network 4 by wire or by wireless connection.

[0051] The camera interface 108 transmits and receives data to and from the camera 118. The camera interface 108 may be connected to the camera 118 by a wire or by wireless connection.

[0052] In the communication unit 106 and the camera interface 108, wired connections such as USB (Universal Serial Bus) or parallel connections may be used. For wireless connections, Bluetooth®, ZigBee®, or wireless LAN (IEEE 802.11 standard) may be used.

[0053] The input unit 116 accepts user input. The input unit 116 includes, for example, a keyboard, mouse, game controller, buttons, a directional pad, etc. The input unit 116 may also be an interface with a device that accepts user input (for example, a game controller). That is, the device that accepts user input may be located outside the game device 100.

[0054] The display 120 displays images or videos generated by processing by the processor 102. The display 120 includes, for example, an LCD (Liquid Crystal Display) or an organic EL display. The display 120 may be located outside the game device 100. In this case, the game device 100 may include an interface circuit to the display 120.

[0055] Microphone 122 is located on the casing of the game device 100 and collects sound to generate an audio signal. Speaker 124 is located on the casing of the game device 100 and generates sound from the audio signal. At least one of microphone 122 and speaker 124 may be located outside the game device 100.

[0056] The camera 118 generates camera data (moving image data or still image data) of the subject in accordance with commands from the game device 100 or according to predetermined conditions.

[0057] The camera 118 includes an image sensor, such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. A lens may be placed in front of the image sensor.

[0058] Camera 118 may be, for example, a camera for video calls. Camera 118 may be a dedicated camera that can only be connected to a specific device, or it may be a general-purpose camera that can be connected to any device.

[0059] The camera 118 may be connected to the game device 100 by wire or wireless connection, or it may be incorporated into the game device 100. Multiple cameras 118 may be connected to the game device 100.

[0060] As an example of an information processing device, a game device 100 was given, but the information processing device may also be, for example, a personal computer, television, smartphone, tablet, etc.

[0061] (b2: Distribution server 200) Figure 3 is a schematic diagram showing an example of the hardware configuration of a distribution server 200 according to this embodiment. Referring to Figure 3, the distribution server 200 includes, for example, one or more processors 202, one or more memories 204, one or more communication units 206, storage 210, and an input unit 216.

[0062] Processor 202 is the processing unit for executing processing in the distribution server 200. Processor 202 includes, for example, a CPU and a GPU. Processor 202 loads programs stored in storage 210 into memory 204 and executes them.

[0063] Memory 204 is a volatile storage medium accessible by the processor 202, and includes, for example, DRAM or SRAM.

[0064] The storage 210 is a non-volatile storage medium accessible by the processor 202, and includes, for example, a hard disk or flash memory. The storage 210 stores, for example, a system program 212 and a distribution program 214.

[0065] The storage 210 is a non-volatile storage medium accessible by the processor 202, and includes, for example, a hard disk or flash memory. The storage 210 stores, for example, a system program 212 and a distribution program 214.

[0066] The system program 212 includes computer-readable instructions for hardware control of the distribution server 200 and for providing a program execution environment.

[0067] The distribution program 214 includes computer-readable instructions for performing relay processing and other operations necessary for distribution, as described later.

[0068] The communication unit 206 sends and receives data to and from the game device 100, the management server 300, etc., via the network 4. The communication unit 206 may be connected to the network 4 by wire or by wireless connection.

[0069] The input unit 216 accepts user input. The input unit 216 includes, for example, a keyboard or mouse.

[0070] (b3: Management Server 300) The hardware configuration example for the management server 300 is the same as the hardware configuration example for the distribution server 200 shown in Figure 3, so a detailed explanation will not be repeated.

[0071] The distribution program stored in the management server 300's storage may include computer-readable instructions for performing the processing necessary for managing distributions, as described later.

[0072] [C. Screen image example] Next, an example of a screen image in the distribution of System 1 according to this embodiment will be described.

[0073] Figure 4 is a schematic diagram showing an example of a screen image 126 displayed on the display 120 of a game device 100-1 according to this embodiment. Figure 4 shows an example of a screen image 126 of a game device 100-1 belonging to the distribution group shown in Figure 1. The game device 100-1 operates as at least a receiving terminal.

[0074] Referring to Figure 4, the screen image 126 includes the main image 130 and the distribution display images 140-1 to 140-4 (hereinafter sometimes collectively referred to as "distribution display images 140").

[0075] Each of the distribution display images 140-2 to 140-4 is output based on image data distributed by a transmitting terminal belonging to the distribution group. In this embodiment, the game device 100-1 itself is the transmitting terminal. Therefore, one distribution display image 140-1 is output based on image data generated by its own device. Each of the other game devices 100-2 to 100-4 displays the same distribution display image 140 as distribution display image 140-1 based on the transmission data 10 distributed by distribution display image 140-1. Note that the display position of the distribution display image 140 may differ for each game device 100. If game device 100-1 is not the transmitting terminal, all of the distribution display images 140-1 to 140-4 are output based on image data distributed by other transmitting terminals.

[0076] Each of the streamed display images 140-1 to 140-4 includes a game image 142, a user icon 144 of the sending terminal user, a username 146 of the sending terminal user, and a camera image 148. Note that at least one of the user icon 144 and username 146 may not be displayed. Also, at least one of the game image 142 and camera image 148 may not be displayed. Furthermore, other images may be displayed in addition to or instead of these images.

[0077] The game image 142 is output based on game image data generated by the game program 180 running on the transmitting terminal. The user of the transmitting terminal may configure whether or not to distribute game image data. The game image 142 is not limited to images generated by the running game program, but may be any image generated on the transmitting terminal. For example, the game image 142 may be an image showing a menu screen displayed on the transmitting terminal's display 120, or an image generated by the transmitting terminal's distribution program 114.

[0078] The user icon 144 may be pre-set or generated by the user of the sending terminal. The username 146 is displayed based on the user information of the sending terminal (e.g., the user account name). The image to be displayed as the user icon 144 and the text to be displayed as the username 146 may be received by the game device 100, or the game device 100 may generate them based on attribute information of user accounts participating in the distribution group.

[0079] The camera image 148 is based on captured image data generated by the camera 118 connected to the transmitting terminal. The user of the transmitting terminal may set whether or not to distribute the captured image data. For example, if the user of the transmitting terminal has not permitted the distribution of captured image data, a predetermined default image (e.g., a black image) may be displayed as the camera image 148. Permission to distribute captured image data on the transmitting terminal may be set by the user using the transmitting terminal, by another user (e.g., a parent or guardian), or by the transmitting terminal or the running application. If the camera 118 is not connected to the transmitting terminal, nothing will be displayed in the area where the camera image 148 is displayed, and the background game image 142 may be displayed. The process for displaying the camera image 148 will be described later.

[0080] The main image 130 includes a game image 132. The game image 132 may be output based on game image data generated by a game program 180 executed in the game device 100 (the device itself). In this case, the user of the game device 100 can enjoy the game while viewing the game status (game image 142) of other players on the transmitting terminal.

[0081] The resolution (or number of pixels) or frame rate of the game image 132 included in the main image 130 may be higher than that of the game image 142 included in the distribution display image 140.

[0082] [D. Camera image 148] Next, the camera image 148 in the distributed display image 140 will be described. The camera image 148 is generated based on the captured image data included in the transmitted data 10.

[0083] Figure 5 is a schematic diagram showing an example of a display mode for a camera image 148 displayed on the display 120 of a game device 100 according to this embodiment. In Figure 5, the area showing the game image 142 is shown with hatching to clarify the display mode of the camera image 148. In this embodiment, the display mode of the camera image 148 is set on the transmitting terminal side, but it may also be set on the receiving terminal side.

[0084] Figure 5(A) shows an example of the face mode display 148A. The face mode display 148A includes an image of a predetermined shape from an image of the user, with the user's face substantially at the center. This makes it easier for the user of the receiving terminal to check the player's facial expressions during the game. The predetermined shape may be a circle, an ellipse, or a polygon. Based on the position of the person's face, an image of the person's face and the area around it is superimposed on the game image 142.

[0085] Figure 5(B) shows an example of the contour cutting mode display 148B. . ring The outline cropping mode display 148B includes an image cropped according to the contours of the human body.

[0086] Figure 5(C) shows an example of the unprocessed mode display 148C. The unprocessed mode display 148C includes the image captured by the camera 118 of the transmitting terminal as is. Therefore, the captured image may include objects in the background of the user in addition to the user as the subject. The image captured by the camera 118 of the transmitting terminal is superimposed on the game image 142.

[0087] Figure 5(D) shows an example of the background blur mode display 148D. The background blur mode display 148D includes an image captured by the camera 118 of the transmitting terminal in which areas other than the human body are blurred.

[0088] [E. Example of image data generation process] Next, we will explain an example of the image data generation process used for face mode display 148A.

[0089] The transmitting terminal processes the camera data output by the camera 118 to generate captured image data. The image size of the captured image data may be predetermined. When generating the captured image data, the transmitting terminal may identify a part of the human body included in the image (camera data) captured by the camera 118. For example, as shown in Figure 5(A), the "part of the human body" may be a person's face.

[0090] When the face mode display 148A shown in Figure 5(A) is selected, the transmitting terminal detects the faces of people included in the captured image. Face detection may be performed using known face recognition techniques. The transmitting terminal extracts partial image data encompassing the detected faces of people from the camera data and generates captured image data 111 from the extracted partial image data.

[0091] Figure 6 is a diagram illustrating an example of the generation process of captured image data 111 in the transmission terminal of System 1 according to this embodiment.

[0092] Figure 6(A) shows an example in which a person's face is included within the field of view of camera 118. The transmitting terminal detects the person's face included in the camera data 119 output by camera 118 and determines the region 136 containing the detected person's face. The transmitting terminal may adjust (reduce / enlarge) the image of the determined region 136 to match a preset image size of the captured image data 111.

[0093] Figure 6(B) shows another example in which a person's face is included within the field of view of camera 118. Similar to Figure 6(A), the transmitting terminal detects the person's face in the camera data 119 and determines the region 136 containing the detected person's face. The transmitting terminal enlarges the determined region 136 to a predetermined image size to generate the captured image data 111. However, in the example shown in Figure 6(B), the person is located far from camera 118, so the region 136 is small compared to the image size of camera data 119. Therefore, the region 136 must be enlarged, which may degrade the image quality of the person's face included in the captured image data 111.

[0094] In System 1 according to this embodiment, the transmission data transmitted by the transmitting terminal may include information that allows for the determination of a decrease in the image quality of a person's face in the captured image data 111. This allows the receiving terminal to determine whether or not to generate a camera image 148 from the received captured image data 111. For example, if the image quality of a person's face is degraded, the receiving terminal may notify the transmitting terminal that the camera image 148 cannot be displayed because the user of the transmitting terminal is too far away from the camera 118.

[0095] Figure 6(C) shows an example where a person's face is not included in the field of view of camera 118. The transmitting terminal cannot detect a person's face in the camera data 119. When detection fails, the transmitting terminal may perform any processing, but for example, a default image (e.g., a black image) is output as the captured image data 111. In the following description, the default image shown in Figure 6(C) will also be referred to as a "mute image" in the sense that the receiving terminal does not display the captured image. A mute image is not limited to an image in which all pixels are black; it can be any image. For example, a mute image may be an image in which all pixels are white.

[0096] Figure 6(D) shows an example of processing when distribution of images captured by camera 118 is not permitted. Since distribution is not permitted, the transmitting terminal outputs a default image as captured image data 111. In this case as well, the default image output as captured image data 111 may be a muted image.

[0097] In other words, if a person's face cannot be detected from the camera data 119, and if the distribution of captured image data 111 is not permitted, a muted image may be output as captured image data 111. Therefore, when a receiving terminal receives transmission data 10 containing a muted image, it may not be able to determine the reason why the muted image was transmitted.

[0098] In System 1 according to this embodiment, the transmission data transmitted by the transmitting terminal may include information indicating whether or not a person's face is included within the field of view of the camera 118. That is, the transmission data may include information indicating whether or not a person's face is included in the captured image. This allows the receiving terminal to determine that a muted image is being transmitted because no person's face is within the field of view of the transmitting terminal's camera 118. In this case, the transmitting terminal may notify the receiving terminal that it cannot display the image of the transmitting terminal's user because no person's face is within the field of view of the transmitting terminal's camera 118.

[0099] When the contour clipping mode display 148B shown in Figure 5(B) is selected, if a person's face cannot be detected from the camera data 119, the area to be clipped cannot be determined, and therefore no valid camera image 148 exists. For this reason, the game image 142 may be displayed without being obstructed by the camera image 148. When the unprocessed mode display 148C shown in Figure 5(C) is selected, the transmitting terminal may output the camera data 119 as is as captured image data 111. When the background blur mode display 148D shown in Figure 5(D) is selected, the transmitting terminal may blur the area other than the human body included in the camera data 119 to generate captured image data 111.

[0100] [F. Transmission Data 10] Next, we will explain the transmission data 10 sent by the transmitting terminal.

[0101] Figure 7 is a schematic diagram showing an example of the data structure of the transmitted data 10 in system 1 according to this embodiment.

[0102] Referring to Figure 7, the transmitted data 10 may be in a data format suitable for streaming, for example. The transmitted data 10 may include, for example, captured image stream data 11 and game image stream data 12.

[0103] The captured image stream data 11 includes a face presence / absence flag 113, a distribution permission / denial flag 115, a magnification factor 117, and captured image data 111.

[0104] The face presence / absence flag 113 is an example of information indicating whether or not a person's face is included in the captured image data 111. For example, if a person's face is included in the captured image data 111, the face presence / absence flag 113 is set to "1", otherwise the face presence / absence flag 113 is set to "0".

[0105] The distribution permission flag 115 is an example of information indicating whether or not distribution of images generated by camera 118 is permitted. For example, if distribution of images generated by camera 118 is permitted, the distribution permission flag 115 is set to "1", otherwise, the distribution permission flag 115 is set to "0".

[0106] The magnification ratio 117 represents the ratio of the size of the captured image data 111 to the size of the region 136 set in the camera data 119. Since the image size of the captured image data 111 is predetermined, the magnification ratio 117 is also an indicator of the size of a person's face (part of the human body) included in the captured image. The magnification ratio 117 is an example of information used to determine whether or not the image quality of a person's face included in the captured image data 111 is degraded. The larger the magnification ratio 117, the lower the image quality of the person's face in the captured image data 111. Thus, the magnification ratio 117 is information that can be used to determine whether or not a person's face is captured in a grainy state.

[0107] The face presence / absence flag 113 and the magnification level 117 may be transmitted only if necessary.

[0108] The captured image data 111 is generated by the image processing described above.

[0109] The game image stream data 12 includes game image data 121 necessary for generating the game image 142 of the distributed display image 140.

[0110] The captured image stream data 11 may be generated in accordance with the H.264 (MPEG-4 AVC) standard. Similarly, the game image stream data 12 may also be generated in accordance with the H.264 standard. That is, the transmitting terminal may transmit stream data in accordance with the H.264 standard that includes captured image data 111 and status data based on the state of the human body in the captured image (face presence / absence flag 113, distribution permission / denial flag 115, and magnification 117). Likewise, the transmitting terminal may transmit stream data in accordance with the H.264 standard that includes game image data 121.

[0111] Stream data conforming to the H.264 standard consists of a sequence of data called NAL (Network Abstraction Layer) units. NAL units can be broadly classified into VCL (Video Coding Layer)-NAL units, which contain video information, and non-VCL-NAL units, which do not contain video information.

[0112] When the captured image stream data 11 is generated in accordance with the H.264 standard, for example, one NAL unit (non-VCL-NAL unit) stores the status data for that frame (face presence / absence flag 113, distribution permission / denial flag 115, and magnification 117) (stream data 11n). The captured image data 111 for one frame is stored in a subsequent NAL unit (VCL-NAL unit) (stream data 11v).

[0113] In other words, the image capture stream data 11 for one frame may consist of stream data 11v, which is a VCL-NAL unit containing image capture data 111, and stream data 11n, which is a non-VCL-NAL unit containing state data. On the other hand, the game image stream data 12 for one frame may consist of a single VCL-NAL unit containing game image data 121. Note that some or all of the state data may be included in the VCL-NAL unit.

[0114] Each of the VCL-NAL units, including the captured image data 111 and the game image data 121, may include either keyframes (frames that have not undergone inter-frame compression) or difference frames (delta frames).

[0115] [G. Example of processing for distribution] Next, we will describe an example of the processing for distribution in the game device 100.

[0116] (g1: Sending terminal) Figure 8 is a schematic diagram showing an example of the software configuration in the transmitting terminal of System 1 according to this embodiment. Figure 8 shows an example of the software configuration focusing on the case where the game device 100 operates as a transmitting terminal, and for the sake of explanation, other processing modules are not shown.

[0117] Referring to Figure 8, the game device 100, which operates as a transmitting terminal, includes a distribution program execution unit 150, a game program execution unit 154, and a transmission unit 156.

[0118] The distribution program execution unit 150 is implemented by executing the distribution program 114 in an environment where the processor 102 of the game device 100 is executing the system program 112. The distribution program execution unit 150 includes a camera image processing unit 151 and a game image processing unit 152.

[0119] The camera image processing unit 151 generates captured image data 111 based on the captured image (camera data 119) from the camera 118. The camera image processing unit 151 generates captured image data 111 depending on whether or not distribution of the captured image data 111 is permitted (distribution permission / denial) and the display mode of the camera image 148.

[0120] As shown in Figure 6(A) above, when the image captured by the camera 118 includes a person's face (an example of a part of the human body), the camera image processing unit 151 generates an image of a part of the captured image, specifically the region 136 that includes the part of the human body, as captured image data 111.

[0121] In order to generate captured image data 111 with a predetermined image size, the camera image processing unit 151 may adjust (reduce / enlarge) the image size of the partial image data corresponding to the region 136 that includes a part of the human body. For example, if the region 136 that includes a part of the human body is smaller than the predetermined image size of the captured image data 111, the camera image processing unit 151 may generate an image of the region 136 that has been stretched as the captured image data 111.

[0122] The position and size of region 136 are determined according to the face of the person being detected. The camera image processing unit 151 generates the captured image data 111 such that, for example, the area of ​​the person's face (part of the human body) within the image of region 136 (captured image data 111) falls within a predetermined proportion. In other words, the camera image processing unit 151 determines the position and size of region 136 so that the area other than the detected person's face does not become too large compared to the person's face.

[0123] The camera image processing unit 151 outputs state data based on the imaging state of the human body in the captured image. More specifically, the camera image processing unit 151 outputs face presence / absence information 153, distribution permission / denial information 155, and magnification 157, which are determined in the process of generating the captured image data 111.

[0124] The facial presence / absence information 153 is as described above. Face presence / absence flag 113 This is information used to determine the value. The face presence / absence information 153 indicates whether or not the captured image data 111 contains a person's face.

[0125] Distribution permission information 155 is as described above. Distribution permission flag 115 This is information used to determine the value.

[0126] The magnification of 157 is information used to determine the value of the magnification of 117 mentioned above.

[0127] The game program execution unit 154 is implemented by executing the game program 180 in an environment where the processor 102 of the game device 100 is executing the system program 112. The game program execution unit 154 advances the game in response to user operations according to computer-readable instructions contained in the game program 180. The game program execution unit 154 outputs output image data 128 as image data of the game being executed. The output image data 128 has the resolution (or number of pixels) and frame rate that are displayed when the user plays the game on the game device 100.

[0128] The game image processing unit 152 generates game image data 121 from the output image data 128. The game image processing unit 152 generates game image data 121, for example, by downsampling or downconverting the output image data 128. The game image processing unit 152 may be implemented using functions provided by the system program 112, etc.

[0129] The transmission unit 156 is implemented by the game device 100's processor 102 executing a system program 112 and utilizing the communication unit 106. At least part of the processing performed by the transmission unit 156 may follow the distribution program 114. The transmission unit 156 generates and transmits captured image stream data 11 based on captured image data 111, face presence / absence information 153, distribution permission / denial information 155, and magnification 157.

[0130] The transmission unit 156 generates and transmits game image stream data 12 based on the game image data 121.

[0131] Figure 9 is a flowchart showing the processing procedure in the transmitting terminal of System 1 according to this embodiment. Each step shown in Figure 9 is realized, for example, by the processor 102 of the game device 100, which operates as a transmitting terminal, executing the system program 112 and the distribution program 114. It is assumed that the processor 102 of the game device 100 is also executing the game program 180.

[0132] The sending terminal is assumed to have performed the necessary processing in advance to join a distribution group, to belong to a certain distribution group, and to be operating as a sending terminal within that distribution group.

[0133] Referring to Figure 9, the transmitting terminal acquires output image data 128 output by the running game program 180 (step S100). The game device 100 generates game image data 121 from the output image data 128 (step S102).

[0134] The transmitting terminal generates game image stream data 12 based on game image data 121 (step S104) and transmits the generated game image stream data 12 (step S106).

[0135] The transmitting terminal determines whether or not it is permitted to distribute the captured image data 111 (step S108).

[0136] If the distribution of the captured image data 111 is not permitted (NO in step S108), the transmitting terminal outputs image data indicating a muted image as the captured image data 111 (step S110), and outputs "0" as the distribution permission information 155, indicating that the distribution of the captured image data 111 is not permitted (step S112). The transmitting terminal does not need to output the face presence / absence information 153 and the magnification 157.

[0137] The condition that the distribution of captured image data 111 is not permitted means that the condition is met when it is selected not to transmit the captured image from camera 118. If the distribution of captured image data 111 is not permitted, the transmitting terminal (camera image processing unit 151) outputs a muted image, which is an example of predetermined image data, as captured image data 111. If the distribution of captured image data 111 is permitted, the conditions described below will not be met, and the transmitting terminal does not need to transmit face presence / absence information 153 indicating that the captured image does not contain any part of a human body.

[0138] If the distribution of the captured image data 111 is permitted (YES in step S108), the transmitting terminal acquires camera data 119 from the camera 118 (step S114).

[0139] The transmitting terminal determines the selected display mode (step S116).

[0140] If face mode is selected (in step S116, "face mode"), the transmitting terminal determines whether or not it can detect a person's face in the camera data 119 (step S118).

[0141] If a person's face can be detected (YES in step S118), the transmitting terminal generates captured image data 111 from the region 136 containing the detected person's face (step S120). The transmitting terminal calculates the magnification ratio when generating captured image data 111 from the partial image data corresponding to region 136 (step S122). The transmitting terminal outputs "1" as face presence / absence information 153 to indicate that a person's face is included, outputs "1" as distribution permission / denial information 155 to indicate that distribution of captured image data 111 is permitted, and outputs the magnification ratio 157 when generating captured image data 111 from camera data 119 (step S124).

[0142] If a person's face cannot be detected (NO in step S118), the transmitting terminal outputs a muted image as captured image data 111 (step S126). The condition that a person's face cannot be detected in the camera data 119 is an example of a condition that is met when the captured image (camera data 119) does not contain any part of a human body. When such a condition is met, the transmitting terminal (camera image processing unit 151) outputs, for example, a muted image as captured image data 111.

[0143] The transmitting terminal outputs "0" as face presence / absence information 153, indicating that the captured image does not contain a person's face; "1" as distribution permission / denial information 155; and an invalid value (e.g., 0% or a negative value) as the magnification factor 157 (step S128). Thus, the status data includes data indicating that the captured image does not contain any part of the human body. Such data can also be interpreted as meaning that no part of the human body can be detected in the captured image.

[0144] If either the no-processing mode, background blur mode, or contour cut-out mode is selected (as indicated in step S116 as "no-processing mode, etc."), the transmitting terminal generates captured image data 111 from the camera data 119 (step S130). The transmitting terminal outputs "1" as the distribution permission information 155 (step S132). Here, the transmitting terminal does not need to output the face presence / absence information 153 and the magnification 157.

[0145] If face mode is selected, the transmitting terminal may transmit information indicating the outline of the human body by some means. Also, if background blur mode is selected, the transmitting terminal blurs the area of ​​the camera data 119 other than the outline of the person.

[0146] The transmitting terminal generates captured image stream data 11 based on the captured image data 111, face presence / absence information 153, distribution permission / denial information 155, and magnification 157 (step S134), and transmits the generated captured image stream data 11 (step S136). Note that if the face presence / absence information 153 and magnification 157 are not output, the transmitting terminal does not need to include the face presence / absence flag 113 and magnification 117 in the captured image stream data 11.

[0147] The transmitting terminal determines whether it is still functioning as a transmitting terminal (step S138). If it is still functioning as a transmitting terminal (YES in step S138), the process from step S100 onward is repeated. If it is not still functioning as a transmitting terminal (NO in step S138), the process ends.

[0148] Furthermore, the transmitting terminal may provide information to assist the user depending on the result of the process of generating captured image data 111 from camera data 119.

[0149] For example, if the camera data 119 cannot detect a person's face (NO in step S118), a message such as "Please move your face so that it is included in the detection range" may be displayed on the display 120.

[0150] Furthermore, if the magnification 117 calculated in step S122 exceeds a predetermined threshold, a message such as "Please move closer to the camera" may be displayed on the display 120 of the transmitting terminal.

[0151] (g2: receiving terminal) Figure 10 is a schematic diagram showing an example of the software configuration in a receiving terminal of System 1 according to this embodiment. Figure 10 shows an example of the software configuration focusing on the case where the game device 100 operates as a receiving terminal, and for the sake of explanation, other processing modules are not shown.

[0152] Referring to Figure 10, the game device 100, which operates as a receiving terminal, includes a distribution program execution unit 150, a game program execution unit 154, and a receiving unit 160.

[0153] The receiving unit 160 is implemented using the communication unit 106 by the processor 102 of the game device 100 executing the system program 112. At least part of the processing performed by the receiving unit 160 may follow the distribution program 114. The receiving unit 160 receives captured image stream data 11 and game image stream data 12 from each of one or more transmitting terminals.

[0154] The distribution program execution unit 150 is implemented by executing the distribution program 114 in an environment where the processor 102 of the game device 100 is executing the system program 112. The distribution program execution unit 150 includes an image capture generation unit 162, a distribution display image generation unit 164, and a screen image generation unit 166.

[0155] The image capture generation unit 162 generates camera image data 171 based on the captured image data 111 included in the captured image stream data 11. The image capture generation unit 162 determines the process for generating camera image data 171 based on the face presence / absence flag 113, the distribution permission / denial flag 115, and the magnification 117. The image capture generation unit 162 may also generate a message 168 based on the face presence / absence flag 113, the distribution permission / denial flag 115, and the magnification 117.

[0156] The distribution display image generation unit 164 generates distribution display image data 173 based on the game image data 121 and camera image data 171 included in the game image stream data 12. The distribution display image generation unit 164 reflects messages 168 in the distribution display image data 173 as needed (for example, messages 149A, 149B, and 149C shown in Figure 12 below).

[0157] The game program execution unit 154 is the same as the game program execution unit 154 shown in Figure 8.

[0158] The screen image generation unit 166 displays the screen image 126 on the display 120 based on the output image data 128 and the distributed display image data 173. The screen image generation unit 166 updates the display content of the main image 130 based on the output image data 128 and updates the display content of the distributed display images 140-1 to 140-4 (see Figure 4) based on the distributed display image data 173.

[0159] Furthermore, if the game device 100 is both a transmitting and receiving terminal, the display content of the distribution display image 140 (for example, the distribution display image 140-1 shown in Figure 4) corresponding to the device may be updated based on the captured image data 111 output by the camera image processing unit 151 (see Figure 8) and the game image data 121 output by the game image processing unit 152 (see Figure 8).

[0160] Alternatively, if the game device 100 is both a transmitting and receiving terminal, it may update the display content of the distribution display image 140 corresponding to its own device based on the captured image stream data 11 and game image stream data 12 that the device transmits to other terminals.

[0161] Figure 11 is a flowchart showing the processing procedure in a receiving terminal of System 1 according to this embodiment. Each step shown in Figure 11 is realized, for example, by the processor 102 of the game device 100, which operates as a receiving terminal, executing the system program 112 and the distribution program 114. It is assumed that the processor 102 of the game device 100 is also executing the game program 180.

[0162] A receiving terminal is assumed to be operating as a receiving terminal within a distribution group by having previously performed the necessary steps to join that distribution group and belonging to that group.

[0163] Referring to Figure 11, the receiving terminal focuses on one of the transmitting terminals (step S200) and receives the captured image stream data 11 and the game image stream data 12 from the focused transmitting terminal (step S202). Note that the reception of the captured image stream data 11 and the reception of the game image stream data 12 may be performed independently of each other.

[0164] The receiving terminal obtains game image data 121 included in the game image stream data 12 from the transmitting terminal of interest (step S204). The receiving terminal determines whether the value of the distribution permission flag 115 included in the captured image stream data 11 from the transmitting terminal of interest is "0" (i.e., a value indicating that distribution of the captured image data 111 is not permitted) (step S206). If the value of the distribution permission flag 115 is "0" (YES in step S206), the receiving terminal determines the camera image data 171 in the case where distribution is not permitted, and also determines a message 168 indicating that distribution is not permitted (for example, "Distribution is not permitted") (step S208).

[0165] If the value of the distribution permission flag 115 is not "0" (NO in step S206), the receiving terminal determines whether the value of the face presence flag 113 is "0" (i.e., a value indicating that the captured image does not contain a person's face) (step S210). If the value of the face presence flag 113 is "0" (YES in step S210), the receiving terminal determines the camera image data 171 in the case where the captured image does not contain a person's face, and also determines a message 168 indicating that the captured image does not contain a person's face (for example, "No face is visible") (step S212).

[0166] If the value of the face presence / absence flag 113 is not "0" (NO in step S210), the receiving terminal determines whether the magnification value 117 exceeds a predetermined threshold (step S214). If the magnification value 117 exceeds a predetermined threshold (YES in step S214), the receiving terminal determines camera image data 171 in which the image quality of the person's face included in the captured image data 111 is degraded, and also determines a message 168 indicating that the image quality of the person's face included in the captured image data 111 is degraded (for example, "too far away") (step S216).

[0167] If the magnification value 117 does not exceed a predetermined threshold (NO in step S214), the receiving terminal decides to use the captured image data 111 as is for camera image data 171 (step S218). At this time, the receiving terminal decides that outputting message 168 is unnecessary. Note that even if the captured image stream data 11 does not contain a valid magnification value 117, it may still be determined that the magnification value 117 does not exceed a predetermined threshold (NO in step S214).

[0168] The receiving terminal updates the distribution display image 140 for the transmitting terminal of interest based on the game image data 121, the determined camera image data 171, and the message 168 (step S220). Note that the message 168 may not exist.

[0169] The receiving terminal determines whether the process of updating the distribution display image 140 has been completed for all transmitting terminals (step S222). If there are any transmitting terminals for which the process of updating the distribution display image 140 has not been completed (NO in step S222), the process from step S200 onward is repeated.

[0170] The receiving terminal updates the main image 130 based on the output image data 128 output by the game program 180 running on its own device (step S224).

[0171] The receiving terminal determines whether its relationship to the distribution group is still ongoing (step S226). If the relationship to the distribution group is still ongoing (YES in step S226), the process from step S200 onward is repeated. If the relationship to the distribution group has ended (NO in step S226), the process ends.

[0172] Furthermore, if the receiving terminal is also a transmitting terminal, it may generate its own distributed display image 140 based on camera data 119 output by camera 118 and game images generated by game program 180 running on its device, rather than on captured image stream data 11 and game image stream data 12 transmitted by the receiving terminal.

[0173] Figure 12 is a diagram illustrating an example of a user interface screen in a receiving terminal of System 1 according to this embodiment.

[0174] Referring to Figure 12(A), if the receiving terminal determines, based on the value of the face presence / absence flag 113 included in the transmitted data 10, that the image captured from the transmitting terminal does not contain a person's face (after step S212 is executed), it may display a message 149A such as "No face is visible" associated with the camera image 148.

[0175] Referring to Figure 12(B), if the receiving terminal determines, based on the magnification value 117 included in the transmitted data 10, that the image quality of the person's face included in the captured image data 111 of the transmitted data 10 has deteriorated (after step S216 is performed), it may display a message 149B such as "Too far away" associated with the camera image 148.

[0176] Referring to Figure 12(C), if the receiving terminal determines that the distribution of the captured image data 111 is not permitted based on the value of the distribution permission flag 115 included in the transmitted data 10 (after step S208 is executed), it may display a message 149C such as "Distribution is not permitted" associated with the camera image 148.

[0177] When displaying messages 149A to 149C, etc., the camera image 148 generated based on the captured image data 111 may be displayed. For example, Figure 12(A) corresponds to the case where the captured image data 111 includes a muted image, and Figure 12(B) corresponds to the case where the captured image data 111 is a low-quality image. In Figure 12(B), the muted image may be displayed as the camera image 148 instead of the captured image data 111. Furthermore, in any of Figures 12(A) to 12(C), any image generated by the receiving terminal may be displayed as the camera image 148. For example, as shown in Figure 12(C), an image reflecting the user information (e.g., user account name) operating the transmitting terminal may be displayed as the camera image 148. Furthermore, the camera image 148 may be hidden. In this case, Camera image 148 Game image 142 is displayed without being obstructed.

[0178] Whether or not camera image 148 is displayed, and the type of image to be displayed as camera image 148, may be determined or changed according to the operation of the user of the receiving terminal.

[0179] In this way, the receiving terminal generates a distribution display image 140 for display on the display 120, which includes a camera image 148 based on the captured image data 111, or a camera image 148 based on other image data instead of the captured image data 111, based on the status data. The receiving terminal displays a screen image 126 which includes one or more of the generated distribution display images 140.

[0180] Messages 149A to 149C may be displayed at all times while the distribution display image 140 is being displayed, or they may be displayed only for a period of time when predetermined conditions are met. For example, messages 149A to 149C may be displayed in response to a selection operation of the camera image 148 by the user of the receiving terminal.

[0181] Whether or not messages 149A to 149C are displayed, and the conditions for displaying messages 149A to 149C, may be determined or changed according to the operation of the user of the receiving terminal.

[0182] [H. Variant] In the above explanation, a person's face was described as an example of a "part of the human body," but the entire body may be used as the detection target, not just the face. The captured image may include not only a part of the human body (such as a person's face or entire body) but also objects in the background, but the captured image data may be output as a cutout image that does not include the background (see Figure 5(B), etc.).

[0183] In the above description, the captured image stream data 11 (see Figure 7) includes (face presence / absence flag 113, distribution permission / denial flag 115, and magnification 117). However, some of these may be omitted, or other data may be included in place of or in addition to these as data indicating the imaging state of the human body. For example, if the captured image data 111 contains a person's face, the face presence / absence flag 113 may be transmitted, and if the captured image data 111 contains a person's face, the magnification 117 may be transmitted. In this case, the stream data 11n includes only one of the distribution permission / denial flag 115 and the magnification 117.

[0184] The above explanation shows an example where all state data is stored in stream data 11v (Figure 7), but at least some of the state data may be stored in different NAL units. For example, the face presence / absence flag 113 may be stored in one NAL unit, and the distribution permission / denial flag 115 may be stored in another NAL unit. By storing the face presence / absence flag 113 and the distribution permission / denial flag 115 in different NAL units, the transmission timing of each flag can be flexibly controlled.

[0185] The face presence / absence flag 113 and the magnification 117 may be transmitted only when face mode is selected. However, at least one of the face presence / absence flag 113 and the magnification 117 may be transmitted even when other modes are selected.

[0186] At least some of the status data (face presence / absence flag 113, distribution permission / denial flag 115, and magnification 117) included in the captured image stream data 11 may be transmitted in a different way from the captured image stream data 11 (for example, in another stream data). For example, the face presence / absence flag 113 may be transmitted by the captured image stream data 11 (stream data 11n), and the distribution permission / denial flag 115 may be transmitted in a different stream. The captured image data 111 needs to be transmitted frame by frame, but the status data may not need to be transmitted frame by frame. Also, at least some of the status data may not need to be synchronized with the captured image data 111.

[0187] In the above explanation, the face presence / absence information 153 (and the face presence / absence flag 113 generated based on the face presence / absence information 153) is a binary value indicating whether or not the captured image contains a part of the human body (e.g., a person's face). Therefore, it can be data indicating that the captured image does not contain a part of the human body, or data indicating that the captured image contains a part of the human body. Instead of using such a binary value, information may be added only when the captured image does not contain a part of the human body, or conversely, only when the captured image contains a part of the human body. In the latter case, the receiving terminal can determine that the captured image does not contain a part of the human body because no information has been added.

[0188] Furthermore, instead of the face presence / absence information 153 (and the face presence / absence flag 113 of the captured image stream data 11) may be used as information indicating whether or not the captured image contains a part of the human body (a person's face). For example, if the magnification is set to "0" or a value that is not normally used (e.g., a negative value), the receiving terminal may determine that the captured image does not contain a part of the human body.

[0189] In the explanation above, a magnification of 157 (magnification of 117 for the captured image stream data 11) was used as an example to show the size of a person's face included in the captured image. Such data is information that can be used to determine whether or not the image of a person's face is blurry. For example, the larger the magnification of 157, the blurrier the image (low quality image) can be considered to be.

[0190] The transmitting terminal may transmit information (flags) indicating whether the magnification 157 exceeds a predetermined threshold, rather than the magnification 157 itself. Alternatively, the transmitting terminal may transmit the image size of the partial image data corresponding to region 136, or the size of the region of the detected person's face.

[0191] If a person's face is not present in the captured image, the transmitting terminal may, instead of sending information indicating that a person's face is not present in the captured image, generate and send an image containing a message such as "No face is visible." However, sending information indicating that a person's face is not present in the captured image allows for greater processing flexibility on the receiving terminal, for example. For instance, the receiving terminal may display a screen image containing a mute image, and only after the mute image has been selected may it display a message or pop-up such as "No face is visible." Alternatively, the receiving terminal may display information indicating that a person's face is not present in the captured image at a different location than the displayed mute image.

[0192] The above explanation illustrates an example of a process for detecting whether or not a part of the human body is included in the captured image when the face mode display 148A shown in Figure 5(A) is selected. In addition to the face mode display 148A shown in Figure 5(A), the system may also detect whether or not a part of the human body is included in the captured image and transmit the detection result when the contour cropping mode display 148B shown in Figure 5(B) is selected.

[0193] If the transmitting terminal cannot detect a person's face, it may notify the user to move their face within the camera's detection range.

[0194] The above description shows an example of superimposing a camera image 148 based on captured image data 111 transmitted by the transmitting terminal onto a game image. However, the camera image 148 based on the captured image data 111 may be used in any way. For example, the camera image 148 is executed at the receiving terminal. app The camera image 148 may be superimposed on any image output by the application program. Furthermore, the camera image 148 may be superimposed on the main image 130. Additionally, the camera image 148 may be superimposed on any image output by the receiving terminal (for example, a still image of the background).

[0195] In System 1 according to this embodiment, the transmitting terminal may distribute not only images of its own device but also audio spoken by the user. The audio may be distributed together with the images, or it may be distributed as separate stream data.

[0196] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims rather than by the foregoing description, and all modifications within the meaning and scope equivalent to the claims are intended to be included. [Explanation of Symbols]

[0197] 1 System, 4 Network, 10 Transmission Data, 11 Captured Image Stream Data, 11n,11v Stream Data, 12 Game Image Stream Data, 100 Game Device, 102,202 Processor, 104,204 Memory, 106,206 Communication Unit, 108 Camera Interface, 110,210 Storage, 111 Captured Image Data, 112,212 System Program, 113 Face Presence / Absence Flag, 114,214 Distribution Program, 115 Distribution Permission / Rejection Flag, 116,216 Input Unit, 117,157 Magnification, 118 Camera, 119 Camera Data, 120 Display, 121 Game Image Data, 122 Microphone, 124 Speaker, 126 Screen Image, 128 Output Image Data, 130 Main Image, 132,142 Game Image, 136 Area, 140 Distribution Display Image, 144 User icon, 146 User name, 148 Camera image, 148A Face mode display, 148B Outline cutout mode display, 148C Unprocessed mode display, 148D Background blur mode display, 149A, 149B, 149C, 168 Message, 150 Distribution program execution unit, 151 Camera image processing unit, 152 Game image processing unit, 153 Face presence / absence information, 154 Game program execution unit, 155 Distribution permission / denial information, 156 Transmission unit, 160 Receiving unit, 162 Captured image generation unit, 164 Distribution display image generation unit, 166 Screen image generation unit, 170 Group information, 171 Camera image data, 173 Distribution display image data, 180 Game program, 200 Distribution server, 300 Management server.

Claims

1. A system comprising at least one transmitting terminal and at least one receiving terminal, The aforementioned transmitting terminal is A first image generation unit that generates first image data, The system includes a transmission unit that transmits the first image data, The first image generation unit generates the first image data as a region of the captured image that includes the part of the human body when the image captured by the camera includes a part of the human body. The transmitting unit transmits, in addition to the first image data, status data based on the imaging state, including whether or not the captured image includes a part of the human body. The aforementioned receiving terminal is A receiving unit that receives the first image data and the status data, A system comprising: a second image generation unit that generates an image for display on a display unit, which includes an image based on the first image data, or an image based on second image data that does not include any part of the human body, instead of the first image data, based on the state data.

2. The first image generation unit generates predetermined image data as the first image data when the first condition that the captured image does not include any part of the human body is met. The system according to claim 1, wherein the state data includes first data indicating that the captured image does not contain any part of a human body.

3. The first image generation unit generates the predetermined image data as the first image data even when a second condition different from the first condition is met. The system according to claim 2, wherein the transmitting unit transmits the first image data but does not transmit the first data when the first condition is not met and the second condition is met.

4. The system according to claim 3, wherein the second condition is that the user has selected not to transmit the image captured by the camera from the transmitting terminal to another terminal.

5. The first image generation unit generates an enlarged image of the region containing the part of the human body in the captured image if the region containing the part of the human body is smaller than a predetermined image size, as the first image data. The system according to claim 1, wherein the state data includes second data indicating the size of a part of the human body included in the captured image.

6. The system according to claim 5, wherein the image size of the first image data is predetermined.

7. The system according to claim 5, wherein the first image generation unit generates the first image data such that the area of ​​a part of the human body in the image of the first image data falls within a predetermined proportion.

8. The system according to claim 5, wherein the second data is the magnification ratio.

9. The system according to any one of claims 1 to 8, wherein the part of the human body is a person's face.

10. The transmitting unit transmits stream data in accordance with the H.264 standard, including the first image data and the status data. The system according to any one of claims 1 to 8, wherein the stream data includes a first NAL unit containing the first image data and a second NAL unit containing the state data.

11. A transmitting terminal connected to one or more receiving terminals via a network, A first image generation unit that generates first image data based on the image captured by the camera, The system includes a transmission unit that transmits the first image data, The first image generation unit generates, when the captured image includes a part of the human body, an image of a part of the captured image that includes the part of the human body, as the first image data. The transmitting unit is a transmitting terminal that transmits, in addition to the first image data, status data based on the imaging state, including whether or not the captured image includes a part of a human body.

12. A receiving terminal connected to one or more transmitting terminals via a network, The aforementioned transmitting terminal is A first image generation unit that generates first image data, The system includes a transmission unit that transmits the first image data, The first image generation unit generates the first image data as a region of the captured image that includes the part of the human body when the image captured by the camera includes a part of the human body. The transmitting unit transmits, in addition to the first image data, status data based on the imaging state, including whether or not the captured image includes a part of the human body. The aforementioned receiving terminal is A receiving unit that receives the first image data and the status data, A receiving terminal comprising: a second image generation unit that generates an image for display on a display unit, which includes an image based on the first image data, or an image based on the second image data that does not include any part of the human body, instead of the first image data, based on the state data.

13. The transmitting terminal includes a step of generating first image data, and when the image captured by the camera includes a part of the human body, an image of a part of the captured image that includes the part of the human body is generated as the first image data. The transmitting terminal transmits state data based on the imaging state, including whether or not the first image data and the captured image include a part of a human body. The receiving terminal receives the first image data and the status data, A method comprising the step of the receiving terminal generating an image for display on a display unit, which includes an image based on the first image data, or an image based on a second image data that does not include any part of the human body, instead of the first image data, based on the status data.

14. On a networked computer, The first image data generation step is performed, and if the camera's captured image includes a part of the human body, an image of a portion of the captured image that includes the part of the human body is generated as the first image data, and further The steps include transmitting the first image data and status data based on the imaging state, including whether or not a part of the human body is included in the captured image, The steps include receiving the first image data and the state data, A program that performs the steps of generating an image for display on a display unit, which includes an image based on the first image data, or, in place of the first image data, an image based on second image data that does not include any part of the human body, based on the state data.

Images