Information processing device, information processing method, and program

Abstract

To provide a device and a method for controlling each of display data and transmission data in an information processing device, and regarding the transmission data, generating and transmitting it according to preset information generated by a user.SOLUTION: An information processing device includes a control unit that controls display data output to a display unit and transmission data transmitted via a communication unit. The control unit generates two types of images having different specifications of a specification for a display image that constitutes the display data and a specification for a transmission image that constitutes the transmission data. The control unit sets, for example, the aspect ratio, resolution, and frame rate of the transmission data in advance, generates the transmission data according to the settings, and transmits it via the communication unit.SELECTED DRAWING: Figure 9

Description

Technical Field

[0005] , , , ,

[0006] , , , , , , , , , , , , ,

[0004] , , , ,

[0003] , ,

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program. Specifically, it relates to an information processing apparatus, an information processing method, and a program for executing and distributing a game application by a smartphone.

Background Art

[0002] In recent years, many users have been enjoying various game applications (game apps) using information processing apparatuses such as smartphones.

[0003] Also, the number of users who perform so-called live distribution of the game being executed to other user terminals is increasing. As a conventional technique disclosed regarding game distribution, for example, there is Patent Document 1 (Japanese Patent Application Laid-Open No. 2018-113514).

[0004] When performing live distribution of a game, the game distributor desires the viewer who is the distribution destination to view an image as intended by the game distributor, but it is not always the case that an image as intended by the distributor is distributed.​​​​​​​​​​​​​​​​​​​​​​​​​​

[0007] This disclosure has been made, for example, in view of the above-mentioned problems, and aims to provide an information processing device, an information processing method, and a program that enable users to easily perform pre-configurations to display the intended streaming image on the viewer's terminal, as well as to check the streaming image, in a configuration for running and distributing a game application using a smartphone. [Means for solving the problem]

[0008] The first aspect of this disclosure is, It has a control unit that controls the display data output to the display unit and the transmission data transmitted via the communication unit. The control unit, The information processing device generates two types of images, one with different specifications for the display image constituting the display data and the other with different specifications for the transmission image constituting the transmission data, displays one of the two generated images on the display unit, and transmits the other via the communication unit.

[0009] Furthermore, the second aspect of this disclosure is, This is an information processing method executed in an information processing device. The information processing device has a control unit that controls the display data output to the display unit and the transmission data transmitted via the communication unit. The control unit, The information processing method involves generating two types of images, one with different specifications for the display image constituting the display data and the other with different specifications for the transmission image constituting the transmission data; displaying one of the two generated images on the display unit; and transmitting the other via the communication unit.

[0010] Furthermore, a third aspect of this disclosure is: This is a program that executes information processing in an information processing device. The information processing device has a control unit that controls the display data output to the display unit and the transmission data transmitted via the communication unit. The program controls the control unit, The program generates two types of images, one with different specifications for the display image constituting the display data and the other with different specifications for the transmission image constituting the transmission data. It then displays one of the two generated images on the display unit and transmits the other via the communication unit.

[0011] The program disclosed herein is a program that can be provided, for example, by a storage medium or communication medium that provides it in a computer-readable format to an information processing device or computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program can be realized on the information processing device or computer system.

[0012] Further purposes, features, and advantages of this disclosure will become apparent from the more detailed descriptions based on the embodiments and accompanying drawings described below. In this specification, a system is a logical combination of multiple devices, and the devices in each configuration are not necessarily located within the same enclosure.

[0013] According to one embodiment of the present disclosure, a device and method are realized in which display data and transmission data are controlled in an information processing device, and the transmission data is generated and transmitted according to pre-configured information generated by the user. Specifically, the system includes a control unit that controls the display data output to the display unit and the transmission data transmitted via the communication unit. The control unit generates two types of images with different specifications: the display image that constitutes the display data and the transmission image that constitutes the transmission data. For example, the aspect ratio, resolution, and frame rate of the transmission data are set in advance, and the transmission data is generated according to these settings and transmitted via the communication unit. These processes enable the control of both display data and transmission data in an information processing device, and realize a device and method that generates and transmits transmission data according to pre-configured information generated by the user. Furthermore, the effects described herein are merely illustrative and not limiting, and additional effects may also exist. [Brief explanation of the drawing]

[0014] [Figure 1] This is a diagram for explaining a configuration example of an information processing system that performs game application execution and distribution processing using an information processing device. [Figure 2] This is a diagram for explaining a configuration example of an information processing system that performs game application execution and distribution processing using an information processing device. [Figure 3] This is a diagram for explaining a series of processing sequences including pre - setting processing before game distribution, game execution, and game distribution processing in an information processing device. [Figure 4] This is a diagram for explaining a series of processing sequences including pre - setting processing before game distribution, game execution, and game distribution processing in an information processing device. [Figure 5] This is a diagram for explaining a series of processing sequences including pre - setting processing before game distribution, game execution, and game distribution processing in an information processing device. [Figure 6] This is a diagram for explaining a series of processing sequences including pre - setting processing before game distribution, game execution, and game distribution processing in an information processing device. [Figure 7] This is a diagram for explaining a series of processing sequences including pre - setting processing before game distribution, game execution, and game distribution processing in an information processing device. [Figure 8] This is a diagram for explaining a series of processing sequences including pre - setting processing before game distribution, game execution, and game distribution processing in an information processing device. [Figure 9] This is a diagram for explaining a configuration example of a pre - data - distribution pre - setting menu (UI) displayed on an information processing device (transmission terminal) used by a distribution user. [Figure 10] This is a diagram for explaining an example of a distributed game image. [Figure 11] This is a diagram for explaining specific examples of a plurality of types of pre - defined distribution image setting data. [Figure 12]This diagram illustrates a specific example of a thumbnail editing screen displayed on an information processing device. [Figure 13] This diagram illustrates a specific example of a distribution image editing screen displayed on an information processing device. [Figure 14] This diagram illustrates a specific example of editing processing using the image editing screen displayed on the information processing device. [Figure 15] This diagram illustrates a specific example of editing processing using the image editing screen displayed on the information processing device. [Figure 16] This diagram illustrates a specific example of editing processing using the image editing screen displayed on the information processing device. [Figure 17] This diagram illustrates a specific example of editing processing using the image editing screen displayed on the information processing device. [Figure 18] This diagram illustrates a specific example of editing processing using the image editing screen displayed on the information processing device. [Figure 19] This diagram illustrates a specific example of editing processing using the image editing screen displayed on the information processing device. [Figure 20] This diagram illustrates a specific example of editing processing using the image editing screen displayed on the information processing device. [Figure 21] This diagram illustrates a specific example of editing processing using the image editing screen displayed on the information processing device. [Figure 22] This diagram illustrates a concrete example of automatically switching between pre-configured landscape-oriented and portrait-oriented images during processing. [Figure 23] This diagram illustrates an example of how to use the privacy settings screen. [Figure 24] This diagram illustrates a specific example of a voice adjustment menu (UI) displayed on an information processing device. [Figure 25] This diagram illustrates a specific example of a data distribution settings menu (UI) displayed on an information processing device. [Figure 26]This diagram illustrates a specific example of a voice adjustment menu (UI) displayed on an information processing device. [Figure 27] This diagram illustrates the process of switching the display image of an information processing device between the currently running game image and a monitoring image (viewing image). [Figure 28] This diagram illustrates the output audio switching process that corresponds to the switching of the display image of an information processing device. [Figure 29] This figure illustrates an example of UI display processing in the information processing device of this disclosure. [Figure 30] This diagram illustrates the process of generating and distributing two types of images with different specifications: one for output to the display unit of an information processing device and another for external distribution. [Figure 31] This diagram illustrates an example of image output processing using a two-screen information processing device. [Figure 32] This diagram illustrates an example of image output processing using a two-screen information processing device. [Figure 33] This diagram illustrates an example of audio output processing using an information processing device with a dual-screen configuration. [Figure 34] This diagram illustrates an example of audio output processing using an information processing device with a dual-screen configuration. [Figure 35] This diagram illustrates an example of audio output processing using an information processing device with a dual-screen configuration. [Figure 36] This diagram illustrates an example of audio output processing using an information processing device with a dual-screen configuration. [Figure 37] This diagram illustrates an example of image output and image distribution processing when using an information processing device with a two-screen configuration. [Figure 38] This diagram illustrates an example of audio output processing using an information processing device with a dual-screen configuration. [Figure 39] This figure illustrates an example of audio output control performed by the information processing device of this disclosure. [Figure 40]This diagram illustrates an example of how to perform the audio output configuration process. [Figure 41] This diagram illustrates an example of how to perform the audio output configuration process. [Figure 42] This figure illustrates a specific example of the process of setting an audio input / output path using the audio input / output path setting unit displayed on the display unit of the information processing device of this disclosure. [Figure 43] This figure illustrates a specific example of the process of setting an audio input / output path using the audio input / output path setting unit displayed on the display unit of the information processing device of this disclosure. [Figure 44] This diagram illustrates an example of the UI display when performing output audio settings processing while the game application is not running. [Figure 45] This diagram illustrates an example of the UI display when performing output audio settings processing while the game application is not running. [Figure 46] This figure illustrates a specific example of processing that applies the output voice control processing performed by the information processing device of this disclosure. [Figure 47] This figure illustrates a specific example of processing that applies the output voice control processing performed by the information processing device of this disclosure. [Figure 48] This figure illustrates a specific example of processing that applies the output voice control processing performed by the information processing device of this disclosure. [Figure 49] This figure illustrates a specific example of processing that applies the output voice control processing performed by the information processing device of this disclosure. [Figure 50] This diagram illustrates an example configuration of the information processing device disclosed herein. [Figure 51] This figure illustrates an example of a hardware configuration for a device that can be used as an information processing device or server according to this disclosure. [Modes for carrying out the invention]

[0015] The details of the information processing device, information processing method, and program described herein will be explained below with reference to the drawings. The explanation will follow the following items. 1. Examples of execution and distribution processing of game applications in information processing equipment. 2. Pre-configuration processing before data distribution, data distribution processing, configuration processing during data distribution, and the series of processing sequences executed in the information processing device of this disclosure. 3. Details of the configuration of the data distribution pre-configuration menu (UI) and the data distribution configuration process using the data distribution pre-configuration menu (UI) 4. Details of the configuration of the data distribution settings menu (UI) and the data distribution settings process using the data distribution settings menu (UI) 5. Other functions of the information processing device disclosed herein 5-1. (1) UI display function for screens other than the game execution screen 5-2.(2) Function to generate images for output from the device and images for distribution as images of different characteristics. 5-3. (3) Processing functions when the information processing device (transmitting terminal) 100 has a dual-screen configuration 6. Regarding the audio output control processing of the information processing device. 7. Examples of Information Processing Device Configurations 8. Examples of hardware configurations for information processing devices and servers 9. Summary of the structure of this disclosure

[0016] [1. Examples of game application execution and distribution processing in information processing equipment] First, we will explain an example of the execution and distribution process of a game application in an information processing device.

[0017] The information processing device disclosed herein is, specifically, a smartphone, a device capable of communicating via a network such as the Internet.

[0018] The information processing device disclosed herein is a device capable of, for example, distributing video over a network, or distributing content such as game content or music content by executing a game application.

[0019] Figure 1 shows an example configuration of an information processing system 10 that utilizes the information processing device of the present disclosure. The information processing device (transmitting terminal) 100 is the terminal of the distribution user (e.g., a game execution player) 20. The distribution user (e.g., a game execution player) 20 uses the information processing device (transmitting terminal) 100 to execute the game application.

[0020] This content, consisting of the game app screen and game app audio (app audio), is distributed to the information processing device (receiving terminal) 200 of the viewing user 30 via a network such as the internet. App sound refers to various sounds generated by an app, such as background music (BGM) or sounds that occur within a game app. For example, the example shown in the diagram is a car racing game app, which includes various sounds such as the sound of car engines, the cheers of the crowd, and the sound of collisions during crashes.

[0021] Furthermore, the user running the game using the information processing device (transmitting terminal) 100, i.e., the streaming user 20, provides commentary on the game in progress. Specifically, the streaming user 20's voice is input via the microphone of the information processing device (transmitting terminal) 100 to provide explanations of the game and its status. The voice of the streaming user 20, along with the app audio mentioned above, is transmitted to the information processing device (receiving terminal) 200 on the viewing user 30's side and played back on the information processing device (receiving terminal) 200.

[0022] Furthermore, viewers 30 can input comments such as messages of support into the information processing device (receiving terminal) 200, and these input comments are transmitted via the network to the information processing device (transmitting terminal) 100 on the distribution user 20's side.

[0023] The information processing device (transmitting terminal) 100 on the distribution user 20 side converts the received comments from the information processing device (receiving terminal) 200 into audio data to generate comment audio, and then synthesizes (mixes) the generated comment audio together with the app audio and the voice of the game user before distributing it.

[0024] Furthermore, viewers 30 may directly input comments such as messages of support via voice input through the microphone of the information processing device (receiving terminal) 200. In this case, these input voice comments are transmitted via the network to the information processing device (transmitting terminal) 100 on the distribution user 20's side.

[0025] The information processing device (transmitting terminal) 100 on the side of the distributing user 20 synthesizes (mixes) the voice comments received from the information processing device (receiving terminal) 200 together with the app's voice and the voice of the game user, and then distributes them.

[0026] Note that the information processing system 10 in Figure 1 is an example of a system configuration in which data is transmitted and received directly between an information processing device (transmitting terminal) 100 and multiple information processing devices (receiving terminals) 200. However, as shown in Figure 2, for example, the system may also be configured to transmit and receive data via a management server 50 on a network.

[0027] In the configuration shown in Figure 2, the information processing device (transmitting terminal) 100 on the distribution user 20 side sends data to the management server 50. The information processing device (receiving terminal) 200 on the viewing user 30 side receives this data from the management server 50 and views it.

[0028] Even in this system configuration, the information processing device (transmitting terminal) 100 on the distribution user 20 side synthesizes audio data from multiple different sound sources, including the game application's execution screen data, application audio such as the game application's background music, the voice of the user running the game, and the commentary audio of the viewing user, and sends it to the management server 50. The information processing device (receiving terminal) 200 on the viewing user 30 side receives this synthesized audio data along with image data from the management server 50 and listens to it.

[0029] [2. Pre-data distribution setup process, data distribution process, data distribution setup process, and the series of processing sequences executed in the information processing device of this disclosure.] Next, we will describe the pre-data distribution setup process, data distribution process, and data distribution setup process performed in the information processing device of this disclosure, and the series of processing sequences.

[0030] Referring to Figures 3 to 8, the pre-data distribution setup process, data distribution process, and in-data distribution setup process executed by the information processing device 100, which is the transmitting terminal on the distribution user 20 side that runs and distributes the game, will be explained.

[0031] Figures 3 to 8 show the processes executed by the information processing device 100 (transmitting terminal) on the distribution user 20 side, sequentially as time-series processes (S01 to S11) as follows. (S01) Launch the game app (S02) Perform a user operation to display the main menu (UI: User Interface). (S03) The main menu (UI) is displayed. (S04) Operate (tap) the Record & Stream icon (Rec&stream) in the main menu. (S05) The pre-configuration menu (UI) screen for data distribution is displayed. (S06) After configuring the pre-distribution settings on the data distribution pre-configuration menu (UI) screen, operate the Complete (Ready) icon.

[0032] (S07) The final check screen before broadcasting begins is displayed. (S08) On the final check screen before starting the broadcast, operate the "Start Broadcasting (Go Live)" icon. (S09) The screen switches to the game execution screen and the broadcast begins. (S10) During game execution and distribution processing, perform the operation to display the data distribution settings menu (UI). (S11) The settings menu (UI) for data distribution corresponding to the game execution distribution period is displayed.

[0033] The following explains each of these processes in turn. (Step S01) First, in step S01 shown in Figure 3, the distribution user 20 launches the game application on the information processing device (transmitting terminal) 100.

[0034] (Step S02) Next, in step S01, the distribution user 20 performs a user operation on the information processing device 100 to display the main menu (UI: user interface).

[0035] For example, as shown in Figure 3(S02), by performing a flick operation, sliding a finger from the top to the bottom of the screen of the information processing device 100, the main menu is gradually pulled out from the top and displayed, as shown in Figure 4(S02b). Note that the flick gesture is just one example of a user action used to display the main menu; other user actions may also be used.

[0036] (S03) Upon completion of a flick operation, where a finger is slid from the top to the bottom of the screen of the information processing device 100, the main menu (UI) 101 is displayed on the display unit of the information processing device 100, as shown in step (S03) in the lower right of Figure 4. Main menu 101 contains various operation icons.

[0037] As shown in Figure 4, the main menu 101 contains several icons for Game mode, Focus settings, Display & Sound settings, Multitasking, Screen shot, and Record & Stream.

[0038] By interacting with (tapping) any of these icons, users can perform actions associated with each icon, such as executing the corresponding process or displaying a separate menu associated with that icon. The icons shown in the diagram are just examples; the main menu can display a variety of other icons, such as various control icons specific to a game application.

[0039] (Step S04) Step S04 shown in Figure 5 is an icon operation step that causes the distribution user 20 to perform the processing of the present disclosure.

[0040] In other words, this indicates that the streaming user 20 selected and operated (tapped) the Record & Stream icon (Rec&stream) displayed in the main menu 101.

[0041] (Step S05) In step S04, when the distribution user 20 operates (tap) the Record & Stream icon (Rec&stream) displayed on the main menu 101, the data distribution pre-configuration menu (UI) (user interface) 102 shown at the bottom of Figure 5 is displayed.

[0042] This data distribution pre-configuration menu (UI) 102 is a user interface (UI) that allows the distribution user 20 to configure various settings related to the distribution data, such as the settings for the game images to be distributed, before distributing a game. Details of this data distribution pre-configuration menu (UI) 102 and the pre-configuration process using this UI will be explained later.

[0043] (Step S06) Once the distribution user 20 has completed all settings, such as the image settings for the game to be distributed, using the data distribution pre-configuration menu (UI) 102 shown in Figure 5, the distribution user 20 operates the setup complete (Ready) icon in the data distribution pre-configuration menu (UI) 102, as shown in (step S06) of Figure 7.

[0044] (Step S07) In step S06, when the distribution user 20 operates the "Ready" icon in the data distribution pre-configuration menu (UI) 102, the "Final check screen before distribution" as shown at the bottom of Figure 6 is displayed.

[0045] The "Final Check Screen Before Distribution Starts" displays an example of a game image for distribution generated by the distribution user 20 using the data distribution pre-configuration menu (UI) 102 shown in Figure 5.

[0046] Streaming user 20 reviews this image, and if they agree, they operate the "Go Live" control panel at the bottom of the screen. On the other hand, if you are not satisfied with the displayed game image or wish to make other corrections, operate the "BACK" control. By operating this "BACK" control, you can display the data distribution pre-configuration menu (UI) 102 shown in Figure 5 (S05) again, and make various changes to the distribution data.

[0047] (Step S08) Once the streaming user 20 understands the example of the game image displayed on the "Final Check Before Streaming" screen, the streaming user 20 operates the "Go Live" icon on the final check screen, as shown in Figure 7(S08).

[0048] (Step S09) When the streaming user 20 operates the "Go Live" icon on the final streaming start check screen, as shown in Figure 7(S08), the display data on the display unit of the information processing device 100 switches to the game execution screen, as shown in Figure 7(S09), and streaming begins.

[0049] At this time, the image (distributed game image) distributed from the information processing device 100 is the distributed game image generated by the distribution user 20 using the data distribution pre-configuration menu (UI) (user interface) 102. In other words, an image following the example of the distributed game image displayed on the "final check screen before distribution start" shown in Figure 6 will be transmitted.

[0050] (Step S10) Subsequently, the streaming user 20 uses the information processing device 100 to run the game application. The live streaming of the game is performed in parallel with the execution of this game application.

[0051] During the game's execution and distribution period, the distribution user 20 can operate the information processing device 100 to display the data distribution settings menu (UI). Step (S10) in Figure 8 illustrates this user operation example.

[0052] The data distribution user 20 performs a flick gesture, sliding their finger from the top to the bottom of the screen of the information processing device 100, as shown in Figure 8 (S10). This gesture displays the data distribution settings menu (UI). Note that the flick gesture is just one example of a user operation used to display the data distribution settings menu (UI), and other user operations may also be used.

[0053] (Step S11) In step S10, the distribution user 20 performs an operation on the information processing device 100 to display the data distribution settings menu (UI), such as a flick operation, which displays the data distribution settings menu (UI) 103 corresponding to the game execution distribution period, as shown in Figure 8 (S11).

[0054] This data distribution settings menu (UI) 103 includes an operation section for making changes to the data being distributed, such as audio adjustment processing, and the distribution user 20 can make settings using this operation section.

[0055] Additionally, the data distribution settings menu (UI) 103 includes a chat output area, where chat information from the broadcasting user 20 and the viewing users 30 who are watching the live-streamed game is displayed. The detailed configuration and usage examples of this data distribution settings menu (UI) 103 will be explained later.

[0056] [3. Details of the configuration of the data distribution pre-configuration menu (UI) and the data distribution configuration process using the data distribution pre-configuration menu (UI)] Next, we will explain the configuration of the data distribution pre-configuration menu (UI) and the details of the data distribution configuration process using the data distribution pre-configuration menu (UI).

[0057] As explained earlier with reference to Figure 5, in step S04 shown in Figure 5, when the distribution user 20 operates (tap) the Record & Stream icon (Rec&stream) displayed in the main menu 101, the data distribution pre-configuration menu (UI) (user interface) 102 shown at the bottom of Figure 5 is displayed.

[0058] This data distribution pre-configuration menu (UI) (user interface) 102 is a UI that allows the distribution user 20 to configure various settings related to the distribution data, such as the settings for the game images to be distributed, before distributing the game.

[0059] The following describes the detailed configuration of the data distribution pre-configuration menu (UI) (user interface) 102 and the details of the data distribution configuration process using the data distribution pre-configuration menu (UI) 102.

[0060] Figure 9 shows an example of a data distribution pre-configuration menu (UI) 102 displayed on the information processing device (transmission terminal) 100 used by the distribution user 20. Note that the data distribution pre-configuration menu (UI) 102 shown in Figure 9 can be displayed, for example, by selecting the distribution pre-configuration menu (UI) selection section 120 in the upper left corner, and the icons on either side of it are selected to display different menus (UI).

[0061] As shown in Figure 9, the data distribution pre-configuration menu (UI) 102 displays the following user-operable processing units. (a) Title Editorial Department 121 (b) Display Information Editorial Department 122 (c) Resolution and frame rate setting section 123 (d) Viewing permission target setting unit 124 (e) Stream delay setting unit 125 (f) Thumbnail editing, display section 126 (g) Distribution image editing, display unit 127 (h) Privacy settings screen editing, display section 128 (i) Microphone setting unit 129 (j) Volume setting unit 130 (k) Extension settings UI transition section 131

[0062] "(a) Title Editing Department 121" is an area for users 20 to enter and edit the titles of games they distribute. Streaming user 20 will be able to freely set the title of the game they are streaming. This title will be distributed, for example, along with the live stream images. An example of the game images being streamed is shown in Figure 10.

[0063] As shown in Figure 10, the distributed image includes the title 141 and display information 142, along with an image of the game application being run by the distribution user 20. The display information 142 shown in Figure 10 is an example of display information consisting of, for example, the type and name of the distribution terminal currently performing game distribution. Note that the title 141 and the display information 142 can be freely set by the distribution user 20.

[0064] The information processing device (receiving terminal) 200 on the viewing user 30 side can include not only the game application image being run by the distribution user 20, but also the title 141 and display information 142 set by the distribution user 20.

[0065] Note that the layout of the distributed image shown in Figure 10 is just one example, and the layout of the distributed image can be changed and edited to various different layouts. This layout editing will be explained later.

[0066] "(b) Display Information Editing Section 122" is an area for editing the data to be displayed in the display area of ​​the display information 142, as explained with reference to Figure 10. The streaming user 20 will be able to freely configure the display information that will be shown along with the live game image being streamed. For example, as explained with reference to Figure 10, display information is generated, such as the type and name of the streaming terminal currently performing game streaming. As mentioned above, the streaming user 20 can freely configure the display information.

[0067] The "(c) Resolution and Frame Rate Setting Section 123" is the area where the resolution, frame rate, and aspect ratio of the live streaming game image are set. The user can select one setting from several predefined types of streaming image setting data.

[0068] Specifically, you can select and configure one of the following five settings ((1) to (5)) as shown in Figure 11. (1) 1080p60fps = Aspect ratio (16:9), Resolution (1920×1080), Frame rate (60fps) (2) 1080p30fps = Aspect ratio (16:9), Resolution (1920×1080), Frame rate (30fps) (3) 720p60fps = Aspect ratio (16:9), Resolution (1280×720), Frame rate (60fps) (4) 720p30fps = Aspect ratio (16:9), Resolution (1280×720), Frame rate (330fps) (5) 480p30fps = Aspect ratio (16:9), Resolution (720×480), Frame rate (30fps)

[0069] The information processing device (transmitting terminal) 100 generates a live streaming image according to the information set by the distribution user 20 in the "(c) resolution and frame rate setting unit 123," that is, a live streaming image having the resolution, frame rate, and aspect ratio set by the user, and transmits it via the transmission unit.

[0070] Returning to Figure 9, we will continue the explanation of each component of the data distribution pre-configuration menu (UI). "(d) Viewing permission target setting section 124" is an area that records information about the restrictions on which viewers can view the live streaming image. Specifically, it is possible to set the content to allow viewing without restrictions (Public), to allow viewing only to specific users (Unlisted), or to allow viewing only to designated users (Private).

[0071] The "(e) Stream delay setting section 125" is an area for recording delay tolerance information for the live streaming data stream to be delivered, and can be set to one of three types: Normal, Low, or Ultra-low.

[0072] "(f) Thumbnail editing and display section 126" is the area for editing thumbnails to be displayed on the list of distributed games provided by the distribution management server, for example, and for displaying the thumbnails as a result of the editing.

[0073] If the distribution user 20 wants to edit the thumbnail, they operate (for example, tap) this "(f) Thumbnail editing, display unit 126". This user action will display the thumbnail editing screen. Specifically, a thumbnail editing screen like the one shown in Figure 12 will be displayed.

[0074] Figure 12 shows the following two figures. (1) Example of editing the text area of ​​a thumbnail (2) Example of thumbnail image area editing process

[0075] The text area for the thumbnail is the text area 144 where the game title and other information are displayed, as shown in Figure 12(1). To edit this text area 144, the user operates (taps) the text area editing operation unit 143. This operation enables editing of the text area 144. For example, you can set the characters, size, color, and position to be output in text area 144. For example, you can set and change the text color and background color by selecting your preferred color from color palette 145.

[0076] Furthermore, the thumbnail image area is the image area 147 where the game image is displayed, as shown in Figure 12(2). To edit this image area 147, the user operates (taps) the image area editing operation unit 146. This operation enables editing of the image area 147. For example, you can set the size and position of image area 147.

[0077] Returning to Figure 9, we will continue the explanation of each component of the data distribution pre-configuration menu (UI). "(g) Distribution image editing and display unit 127" is an area that performs editing processing such as layout and display processing of distribution images distributed from the information processing device (transmission terminal) 100.

[0078] If the streaming user 20 wants to edit the streaming image, they operate (for example, tap) this "(g) Streaming Image Editing, Display Unit 127". This user action will display the streaming image editing screen. Specifically, a streaming image editing screen like the one shown in Figure 13 will be displayed.

[0079] Examples of editing processes for streamed images will be explained with reference to Figure 13 and subsequent figures. First, the layout selection process for the distributed image is performed. As shown in Figure 13, multiple different distributed image layouts are displayed on the display unit of the information processing device (transmission terminal) 100.

[0080] In the example shown in Figure 13, six different delivery image layouts are displayed, from (L1) to (L6). The example of the distribution image layout shown in Figure 13 is a landscape layout, meaning it is an example of the distribution image layout when the distribution user 20 holds the information processing device (transmission terminal) 100 in landscape orientation while playing the game and distributing the image. This is displayed when the landscape layout on the left is selected from the landscape / portrait layout selection section 148 at the top of Figure 13.

[0081] If the portrait layout is selected from the landscape / portrait layout selection unit 148, an example of the layout of the streamed image will be displayed when the information processing device (transmitting terminal) 100 is held in landscape (landscape) orientation while playing the game and streaming. An example of the portrait layout will be described later. The layouts (L1) to (L6) shown in Figure 13 are all image data in which the game image to be distributed is output along with the game title and display information, as explained earlier with reference to Figure 10.

[0082] (L1) is a layout in which display information is placed at the top of the game image and the game title is placed at the bottom. (L2) is a layout in which the game title is placed at the top of the game image and the display information is placed at the bottom. (L3) is a layout in which small display information is placed at the top of the game image, and the game title is placed at the bottom.

[0083] (L4) is a layout in which small display information is placed on the lower left of the game image, and the game title is placed on the lower right. (L5) is a layout in which the game title is placed on the lower left of the game image, and small display information is placed on the lower right. (L6) is a layout in which only the game title is placed at the bottom of the game image.

[0084] The distribution user 20 can, for example, select one of these distribution image layouts as the distribution image layout.

[0085] For example, when the distribution user 20 selects a layout (L1) as shown in Figure 14 and operates the control panel (Next), the editing screen for the distribution image of the selected layout (L1) is displayed.

[0086] Figure 15 shows an example of the editing screen for the delivered image of the selected layout (L1). Figure 15 shows the following two figures. (1) Example of editing the text area of ​​a distributed image (2) Example of image area editing process for distributed images

[0087] The text area of ​​the streamed image is the text area 151 where the game title and other information are displayed, as shown in Figure 15(1). To edit this text area 151, the user operates (taps) the text area editing operation unit 152. This operation enables editing of the text area 151. For example, you can set the characters, size, color, and position to be output in the text area 151. For example, you can set and change the text color and background color by selecting your preferred color from the color palette 153.

[0088] Furthermore, the image area of ​​the distributed image is the image area 154 where the game image is displayed, as shown in Figure 15(2). To edit this image area 154, the user operates (taps) the image area editing operation unit 155. This operation enables editing of the image area 154. For example, you can set the image area size (154 pixels), position, and other parameters.

[0089] Furthermore, as shown in Figure 16, it is also possible to edit the background area of ​​the streamed image. The background area is the area surrounding the image area where the game image is displayed, as shown in Figure 16, and is the background area 156 consisting of the background portion of the area where text is displayed. To edit this background area 156, the user operates (taps) the background area editing operation unit 157. This operation enables editing of the background area 156. For example, you can set the color and position of background area 156. For example, you can set and change the background color by selecting your preferred color from color palette 153.

[0090] The example of editing the distributed image, as explained with reference to Figures 13 to 16, is an example of the process in which the distribution user 20 edits the settings of the distributed image when playing a game while holding the information processing device (transmission terminal) 100 in landscape orientation.

[0091] In the data distribution pre-configuration menu (UI) shown in Figure 9, under "[g] Distribution Image Editing, Display Unit 127", the distribution user 20 can play the game while holding the information processing device (transmission terminal) 100 in portrait orientation and also edit the settings for the distribution image when distributing data.

[0092] The following describes an example of editing the streamed image when a streaming user 20 plays a game while holding the information processing device (transmitting terminal) 100 in portrait orientation and streaming the game, with reference to Figure 17 and subsequent figures. First, the layout selection process for the distributed image is performed. As shown in Figure 17, multiple different distributed image layouts are displayed on the display unit of the information processing device (transmission terminal) 100.

[0093] Figure 17 shows six types of distributed image layouts (Lv1) to (Lv6) displayed on the information processing device 100 on the right, and enlarged views of layouts (Lv1) to (Lv3) on the left. Figure 18 also shows six types of distributed image layouts (Lv1) to (Lv6) displayed on the information processing device 100 on the left, and enlarged views of layouts (Lv4) to (Lv6) on the right.

[0094] The example of the distribution image layout shown in Figures 17 and 18 is a vertical layout, meaning it's an example of the distribution image layout when the distribution user 20 holds the information processing device (transmission terminal) 100 vertically (portrait orientation) while playing the game and distributing the image. This is displayed when the vertical layout on the right is selected from the horizontal / vertical layout selection section 148 at the top of Figures 17 and 18.

[0095] All layouts (Lv1) to (Lv6) are image data in which the game image to be distributed is output along with the game title and display information, as explained earlier with reference to Figure 10. As explained earlier with reference to Figure 10, the display information can include, for example, the type and name of the streaming terminal currently performing the game stream. However, the display information can be freely configured by the streaming user 20.

[0096] The layouts (Lv1) to (Lv6) shown in Figures 17 and 18 are as follows: (Lv1) is a layout in which the display information is placed on the left side of the game image and the game title is placed on the right side. (Lv2) is a layout in which the game title and display information are placed to the left of the game image. (Lv3) is a layout where the game title and display information are placed to the right of the game image.

[0097] (Lv4) is a layout in which the game title is placed on the left side of the game image and the display information is placed on the right side. (Lv5) is a layout where the game title and display information are placed to the left of the game image. (Lv6) is a layout where the game title and display information are placed to the right of the game image.

[0098] The distribution user 20 can, for example, select one of these distribution image layouts as the distribution image layout. For example, when the distribution user 20 selects a layout (Lv1) as shown in Figure 19 and operates the control panel (Next), the editing screen for the distribution image of the selected layout (Lv1) is displayed.

[0099] Figure 20 shows an example of the editing screen for the delivered image of the selected layout (Lv1). Figure 20 shows the following two figures. (1) Example of editing the text area of ​​a distributed image (2) Example of image area editing process for distributed images

[0100] The text area of ​​the streamed image is the text area 151 where the game title and other information are displayed, as shown in Figure 20(1). To edit this text area 151, the user operates (taps) the text area editing operation unit 152. This operation enables editing of the text area 151. For example, you can set the characters, size, color, and position to be output in the text area 151. For example, you can set and change the text color and background color by selecting your preferred color from the color palette 153.

[0101] Furthermore, the image area of ​​the streamed image is the image area 154 where the game image is displayed, as shown in Figure 20(2). To edit this image area 154, the user operates (taps) the image area editing operation unit 155. This operation enables editing of the image area 154. For example, you can set the size and position of image area 154.

[0102] Furthermore, as shown in Figure 21, it is also possible to edit the background area of ​​the streamed image. The background area is the area surrounding the image area where the game image is displayed, as shown in Figure 21, and is the background area 156 consisting of the background portion of the area where text is displayed. To edit this background area 156, the user operates (taps) the background area editing operation unit 157. This operation enables editing of the background area 156. For example, you can set the color and position of background area 156. For example, you can set and change the background color by selecting your preferred color from color palette 153.

[0103] Furthermore, the text area 151 and background area 156, as explained with reference to Figures 20 and 21, may be configured to display not only text information but also still images and videos. For example, it may be configured to display still images or videos to explain the game's content. Alternatively, the background area 156 may be configured to sequentially switch between multiple still images that explain the game. In this case, various settings are possible, such as setting it so that the next still image is displayed when the user taps the background area 156, or setting it so that the next still image is displayed after a predetermined amount of time has elapsed.

[0104] Thus, the information processing device (transmitting terminal) 100 of this disclosure, with respect to the distributed image, Landscape layout selection and editing process, Vertical layout selection and editing process, This makes it possible to perform these two types of processing separately.

[0105] These processes enable the distribution data layout to be automatically switched to either a pre-configured landscape-oriented or portrait-oriented layout when the distribution user 20 changes the orientation of the information processing device (transmitting terminal) 100 while performing game distribution. Refer to Figure 22 to explain a specific example of the switching process.

[0106] Figure 22 shows the following figures. (a) Distributing user terminal (b) Receiving user terminal All of these graphs show the changes in the displayed image on each user's terminal over time (t1 to t3).

[0107] As shown in Figure 22(a) Time-series data of the distribution user's terminal, the distribution user runs and distributes the game with the information processing device (transmitting terminal) 100 in landscape orientation at time (t1), then switches to holding the information processing device (transmitting terminal) 100 in portrait orientation and runs and distributes the game again at time (t2), and then again runs and distributes the game with the information processing device (transmitting terminal) 100 in landscape orientation at time (t3).

[0108] On the other hand, as shown in Figure 22(b) Time-series data of the receiving user's terminal, the receiving user continuously receives and watches the game with the information processing device (receiving terminal) 200 held horizontally from time (t1) to (t3).

[0109] In such a case, during the time (t1) and time (t3) when the distributing user is running and distributing the game with the information processing device (transmitting terminal) 100 in landscape orientation, the image set by the landscape distribution image layout setting process described with reference to Figures 13 to 16 is distributed and displayed on the receiving user's terminal. In contrast, during the time (t2) when the distributing user is running and distributing the game with the information processing device (transmitting terminal) 100 in portrait orientation, the image set by the portrait orientation distribution image layout setting process, as explained with reference to Figures 17 to 21, is distributed and displayed on the receiving user's terminal.

[0110] In this way, the sending user can deliver images with the layout set by the user in both the case where the information processing device (transmitting terminal) 100 is oriented horizontally and vertically.

[0111] Returning to Figure 9, we will continue the explanation of each component of the data distribution pre-configuration menu (UI). The "(h) Privacy Settings Screen Editing and Display Unit 128" is a processing unit that edits the "Privacy Settings Screen," which is alternative distribution image data to be distributed and displayed to the information processing unit (receiving terminal) of the viewing user 30 when a distribution user 20, who is running and distributing a game using the information processing unit (transmitting terminal) 100, displays privacy data other than the game screen on the display unit of the information processing unit (transmitting terminal) 100.

[0112] For example, a streaming user 20 may use the information processing device (transmitting terminal) 100 to input a credit card number while running and streaming a game. Alternatively, they may activate the camera and display the image captured by the camera. This data is private data and we do not want to distribute it.

[0113] In this manner, if data other than the game execution screen is displayed on the display unit of the information processing device (transmitting terminal) 100, the data processing unit of the information processing device (transmitting terminal) 100 switches the distribution data to a pre-configured "privacy settings screen" and distributes it. The "(h) Privacy Settings Screen Editing and Display Unit 128" is responsible for editing and displaying this "Privacy Settings Screen".

[0114] For example, the screen data shown in the area labeled "(h) Privacy settings screen editing and display section 128" in Figure 9 is an example of a "privacy settings screen".

[0115] Refer to Figure 23 for an example of how to use the "Privacy Settings screen". Figure 23 shows the following diagrams. (a) Distributing user terminal (b) Receiving user terminal All of these graphs show the changes in the displayed image on each user's terminal over time (t1 to t3).

[0116] As shown in Figure 23(a), the time-series data of the streaming user's terminal, the streaming user uses the information processing device (transmitting terminal) 100 to run and stream a game at time (t1), and displays images captured by the camera function of the information processing device (transmitting terminal) 100 at time (t2). Subsequently, at time (t3), the user uses the information processing device (transmitting terminal) 100 again to run and stream a game.

[0117] As shown in Figure 23(b) Time-series data of the receiving user terminal, at time (t1), the receiving user receives and displays the game screen distributed from the information processing device (transmitting terminal) 100. At time (t2), the system receives and displays the "privacy settings screen" distributed from the information processing device (transmitting terminal) 100. Subsequently, at time (t3), the game screen distributed from the information processing device (transmitting terminal) 100 is received and displayed.

[0118] In this manner, if data other than the game screen, such as camera images, is displayed on the display screen of the user's terminal, the data processing unit of the information processing device (transmitting terminal) 100 switches the distribution data to a pre-configured "privacy settings screen" and distributes it. This process prevents various types of private data, such as camera images, credit card numbers, and PINs, from being accidentally distributed and leaked.

[0119] Returning to Figure 9, we will continue the explanation of each component of the data distribution pre-configuration menu (UI). (i) Microphone setting unit 129 and (j) Volume setting unit 130 are processing units that adjust audio data transmitted or output from the information processing device (transmitting terminal) 100, which is the distribution user's terminal.

[0120] By operating (tapping) either "(i) Microphone setting unit 129" or "(j) Volume setting unit 130", the audio adjustment menu (UI) will be displayed. A specific example is shown in Figure 24.

[0121] By operating (tapping) either "(i) Microphone setting unit 129" or "(j) Volume setting unit 130", a sound adjustment menu (UI) such as the one shown in Figure 24 will be displayed. The audio adjustment menu (UI) shown in Figure 24 includes: Voices of 20 streaming users (voice(stream)) Game sound during gameplay (Game(stream)) Media output sound (Media (volume)) A control panel is provided for adjusting the volume of each of these sounds, allowing the user to adjust the volume of each sound by operating this control panel.

[0122] Returning to Figure 9, we will continue the explanation of each component of the data distribution pre-configuration menu (UI). The "(k) Extension Settings UI Transition Unit 131" is an operation unit for displaying the extension settings UI when configuring or adjusting extensions other than the items shown in Figure 9.

[0123] Features that can be configured and adjusted using the extension settings UI include, for example, Power supply settings, Display / hide settings for notification messages, etc. Automatic brightness adjustment ON / OFF, Detailed settings for the privacy display screen, For example, these functions exist, and by operating (tapping) "(k) Extension Settings UI Transition Section 131", a UI for setting and adjusting the above extensions can be displayed. can.

[0124] [4. Details of the configuration of the Data Distribution Settings Menu (UI) and the data distribution setting process using the Data Distribution Settings Menu (UI)] Next, we will explain the structure of the data distribution settings menu (UI) and the details of the data distribution settings process using the data distribution settings menu (UI).

[0125] As explained earlier with reference to Figure 8, in step S10 shown in Figure 8, when the streaming user 20 executes a process to display the data streaming settings menu (UI) during game execution and streaming, the data streaming settings menu (UI) (user interface) 103 shown in step S11 of Figure 8 is displayed.

[0126] This data distribution settings menu (UI) 103 is a user interface (UI) that allows the distribution user 20 to configure various settings related to the distribution data, such as the settings for the game images to be distributed, while the game is running and being distributed.

[0127] The following describes the detailed configuration of the Data Distribution Settings Menu (UI) 103 and the details of the data distribution setting process using the Data Distribution Settings Menu (UI) 103.

[0128] Figure 25 shows an example of a data distribution settings menu (UI) 103 displayed on the information processing device (transmission terminal) 100 used by the distribution user 20.

[0129] As shown in Figure 25, the following user-operable processing units are displayed in the data distribution settings menu (UI) 103. (a) Audio adjustment unit 181 (b) Privacy settings screen adjustment unit 182 (c) UI display setting section 183 (d) Display image switching unit 184

[0130] (a) The audio adjustment unit 181 is a processing unit that adjusts the audio data that the streaming user 20 streams or outputs during game streaming.

[0131] By operating (tapping) "(a) Audio adjustment unit 181", the audio adjustment menu (UI) will be displayed. A specific example is shown in Figure 26. The UI shown in Figure 26 has a configuration that is almost identical to the pre-configuration UI described earlier with reference to Figure 24.

[0132] The audio adjustment menu (UI) shown in Figure 26 includes: Voices of 20 streaming users (voice(stream)) Game sound during gameplay (Game(stream)) Media output sound (Media (volume)) A control panel is provided for adjusting the volume of each of these sounds, allowing the user to adjust the volume of each sound by operating this control panel.

[0133] "(b) Privacy settings screen adjustment unit 182" is an operation unit for setting and adjusting the "privacy settings screen" as explained earlier with reference to Figure 23. By operating (tapping) "(b) Privacy Settings Screen Adjustment Unit 182", a UI (Privacy Settings Screen Adjustment UI) for configuring and adjusting the "Privacy Settings Screen" will be displayed.

[0134] Users can use the privacy settings adjustment UI to perform actions such as the following: Toggle the use of the "Privacy Settings screen" Switching between automatic and manual execution of the "Privacy Settings Screen" distribution process.

[0135] The "(c)UI display setting unit 183" is an operation unit for switching between displaying or hiding the UI while the game is running.

[0136] The "(d) Display Image Switching Unit 184" is an operation unit for switching the image displayed on the display unit of the information processing device (transmitting terminal) 100 during game execution and distribution by the distribution user 20, to either the game image being played on the information processing device (transmitting terminal) 100 or the monitoring image (viewing image) being viewed by the viewing user 30 on the information processing device (receiving terminal) 200.

[0137] Referring to Figure 27, a specific example of the display image switching process using "(d) display image switching unit 184" will be explained.

[0138] Figure 27 shows the following diagrams. (p) Example of game execution screen display (q) Example of the display of the settings menu (UI) during data distribution. (r) Example of monitoring screen (viewing screen) display

[0139] "(p) Example of game execution screen display" is an example in which the distribution user 20 is running a game, and the display unit of the information processing device (transmitting terminal) 100 displays the game screen that the distribution user 20 is running during the distribution.

[0140] "(q) Example of displaying the data distribution settings menu (UI)" is an example in which the distribution user 20 displays the data distribution settings menu (UI) by performing an operation (for example, a flicking motion by sliding a finger) on the game execution screen shown as "(p) Example of displaying the game execution screen".

[0141] When the distribution user 20 operates (tap) "(d) Display image switching unit 184" displayed in the data distribution settings menu (UI), the monitoring screen (viewing screen) is displayed on the display unit of the information processing device (transmission terminal) 100.

[0142] "(r) Example of monitoring screen (viewing screen) display" corresponds to the game screen that the viewing user 30 is viewing on the information processing device (receiving terminal) 200. Please note that this monitoring screen (viewing screen) will be slightly delayed compared to the (p) game execution screen due to communication delays, etc.

[0143] The information processing device (transmitting terminal) 100 of the distribution user 20 retrieves the transmission data of the information processing device (transmitting terminal) 100, i.e., the distribution data such as the distributed image, from the management server 50 shown in Figure 2, for example, and outputs it to the display unit of the information processing device (transmitting terminal) 100.

[0144] By operating (tapping) the left-hand icon shown in "(r) Example of Monitoring Screen (Viewing Screen) Display" in Figure 27, the streaming user 20 can return to the display screen shown in "(p) Example of Game Execution Screen Display" in Figure 27, that is, the game screen that the streaming user 20 is currently running.

[0145] As mentioned above, the monitoring screen (viewing screen) will be slightly delayed compared to the (p) game execution screen due to communication delays, etc., and the game audio will also be delayed. Therefore, if the streaming user 20 outputs audio corresponding to the game screen being played while the monitoring screen (viewing screen) is displayed, a discrepancy will occur between the image and the sound, causing a sense of unease.

[0146] To prevent such a situation, the information processing device (transmitting terminal) 100 of the distribution user 20 also performs output audio switching processing to output audio corresponding to the displayed data for the sound output from the speaker of the information processing device (transmitting terminal) 100.

[0147] In other words, the audio switching process shown in Figure 28 is performed. (p) While displaying a game execution screen corresponding to the example of the game execution screen, the audio corresponding to the game execution screen is output from the speaker of the information processing device (transmitting terminal) 100. On the other hand, while a monitoring screen corresponding to the example of (r) monitoring screen (viewing screen) is displayed, audio corresponding to the monitoring screen is output from the speaker of the information processing device (transmitting terminal) 100.

[0148] [5. Other functions of the information processing device disclosed herein] Next, other functions of the information processing device of this disclosure will be described.

[0149] The following functions of the information processing device (transmitting terminal) 100 used by the distribution user 20 will be explained in order below. (1) UI display function for screens other than the game execution screen (2) A function to generate images for output from the device and images for distribution as images of different characteristics. (3) Processing functions when the information processing device (transmitting terminal) 100 has a dual-screen configuration

[0150] (5-1. (1) UI display function for screens other than the game execution screen) First, let's explain the UI display function for screens other than the game execution screen.

[0151] As explained earlier, the information processing device (transmission terminal) 100 used by the distribution user 20 outputs various menus (UI), such as the main menu (UI) described with reference to Figure 4, the data distribution pre-setting menu (UI) 102 described with reference to Figures 5, 9 to 24, and the data distribution in progress setting menu (UI) 103 described with reference to Figures 8, 25 to 28. These menus (UI) generally appear and become available after launching the game app.

[0152] However, even when the streaming user 20 is not running the game, they may want to perform configuration or modification tasks such as adjusting game settings or streaming settings. For example, there are times when you might want to perform operations such as configuring or changing game settings or streaming settings while chatting with gaming friends using a chat app, or while live streaming is in progress. The information processing device (transmitting terminal) 100 disclosed herein enables the distributing user 20 to display and operate the UI even in such cases.

[0153] Figure 29 shows a specific example of menu (UI) display processing. Figure 29(1) shows an example of menu (UI) display processing. For example, when a streaming user 20 is chatting with gaming friends using a chat application, or when live streaming processing is being performed, they perform a flick action by sliding their finger from the top to the bottom of the screen of the information processing device 100, as shown in Figure 29(1).

[0154] This process displays a menu (UI) as shown in Figure 29(2) UI display example. By operating (tapping) the icons displayed on the menu (UI), various game settings, streaming settings, etc., can be accessed.

[0155] (5-2.(2) Function to generate images for output from the device and images for distribution as images of different characteristics) Next, we will explain the function of the information processing device (transmitting terminal) 100 used by the distribution user 20, which is "(2) function to generate images for output by the device and images for distribution as images of different characteristics."

[0156] The information processing device (transmitting terminal) 100 disclosed herein can generate, output, and distribute two types of images with different specifications: an image for output to the device itself, i.e., an image to be output to the display unit of the information processing device (transmitting terminal) 100, and an image to be distributed externally.

[0157] Please refer to Figure 30 for a specific example of the process. Figure 30 shows, (1) Image data for display on the device (transmitting terminal), (2) Image data for distribution, These two image data settings are shown as examples.

[0158] For example, "(1) Image data for display on the device (transmitting terminal)" is, (a) Aspect ratio = 21:9 (b) Resolution = 4K (3840 x 1644 pixels) (c) Frame rate = 120bps This is an image with these specifications.

[0159] On the other hand, "(2) Image data for distribution" is, for example, (a) Aspect ratio = 16:9 (b) Resolution = 1920 x 1080 pixels) (c) Frame rate = 60bps This is an image with these specifications.

[0160] In addition, various other settings are available for "(2) Image data for distribution". As shown in the figure, (a) Aspect ratio = 16:9 (b) Resolution = 1920 x 1080 pixels) (c) Frame rate = 60bps Images of the above specifications,

[0161] (a) Aspect ratio = 16:9 (b) Resolution = 1920 x 1080 pixels) (c) Frame rate = 60bps The above specifications, including image settings, can be configured.

[0162] The settings for the distributed image can be configured by operating the "(c) Resolution and Frame Rate Setting Section" of the data distribution pre-configuration menu (UI) 102, as explained earlier with reference to Figure 9. For example, images with various specifications (aspect ratio, resolution, frame rate) as explained earlier with reference to Figure 11 can be used.

[0163] Thus, the information processing device (transmitting terminal) 100 of this disclosure can generate, output, and distribute two types of images with different specifications: an image for output by the device itself, i.e., an image to be output to the display unit of the information processing device (transmitting terminal) 100, and an image to be distributed externally.

[0164] (5-3. (3) Processing functions when the information processing device (transmitting terminal) 100 has a dual-screen configuration) Next, we will explain the processing functions when the information processing device (transmitting terminal) 100 has a two-screen configuration.

[0165] Figure 31 shows an example configuration of a two-screen information processing device (transmitting terminal) 100. For example, it is a foldable smartphone with separate display units at the top and bottom.

[0166] By making the information processing device (transmission terminal) 100 used by the distribution user 20 a smartphone with such a dual-screen configuration, it becomes possible to perform display processing as shown in Figure 31, for example. Figure 31 shows the following two image display examples, namely, (A) Example of image display a (B) Example of image display b These two image display examples are shown.

[0167] "(A) Image display example a" displays "(a1) Game execution screen" in the upper display section and "(a2) Monitoring screen" in the lower display section. On the other hand, "(B) Image display example b" displays "(b1) Monitoring screen" in the upper display section and "(b2) Game execution screen" in the lower display section.

[0168] The game execution screen and the monitoring screen are the same as the screens described earlier with reference to Figure 27. That is, the game execution screen is the game screen being played by the streaming user 20. The monitoring screen corresponds to the game screen being viewed by the viewing user 30 on the information processing device (receiving terminal) 200.

[0169] The monitoring screen, for example, displays the distribution data transmitted by the information processing device (transmitting terminal) 100, which is re-acquired from the management server 50 shown in Figure 2.

[0170] A distribution user 20 having an information processing device (transmission terminal) 100 with a two-screen configuration as shown in Figure 31 can freely switch between displaying "(A) Image display example a," which displays "(a1) Game execution screen" on the upper display unit and "(a2) Monitoring screen" on the lower display unit, and "(B) Image display example b," which displays "(b1) Monitoring screen" on the upper display unit and "(b2) Game execution screen" on the lower display unit, as shown in Figure 31.

[0171] The switching process between "(A) Image display example a" and "(B) Image display example b" can be performed, for example, by the user performing a pinch gesture (touching and pinching or spreading two fingers on the upper and lower screens) or a swipe gesture (touching and sliding a finger on the screen), as shown in Figure 32.

[0172] As mentioned above, the monitoring screen (viewing screen) will have a slightly delayed image compared to the game execution screen due to communication delays, etc., and the game audio will also be delayed. Therefore, if the streaming user 20 outputs audio corresponding to the game screen being played while the monitoring screen (viewing screen) is displayed, a discrepancy will occur between the image and the sound, causing a sense of unease.

[0173] To prevent such situations, the streaming user 20 can switch the output sound from the speaker. In other words, the audio switching process shown in Figures 33 and 34 is performed. Figure 33 shows a configuration in which audio corresponding to the game execution screen is output from the speaker of the information processing device (transmitting terminal) 100, and audio output corresponding to the monitoring screen is stopped. On the other hand, Figure 34 shows a setting where audio corresponding to the monitoring screen is output from the speaker of the information processing device (transmitting terminal) 100, and audio output corresponding to the game execution screen is stopped.

[0174] The speaker output switching control of the information processing device (transmitting terminal) 100 is performed, for example, in the following manner. For example, the information processing device (transmitting terminal) 100 detects user operations (taps) on each screen, or operations (taps) on speaker icons that are individually displayed on each screen (though not shown in the diagram), and executes speaker output audio switching processing according to the detected information.

[0175] Alternatively, the system could be configured to output audio from the speaker that corresponds to the image displayed on one of the two screens, for example, the lower screen. For example, as shown in Figure 35, the two-screen information processing device (transmitting terminal) 100 outputs sound corresponding to the image displayed on the lower screen. In Figure 35, the image display example c1 (C1) shows that the image displayed on the lower screen of the two-screen information processing device (transmitting terminal) 100 is the game execution screen, so the game execution sound is output. On the other hand, in Figure 35, (C2) Image display example c2 shows that the image displayed on the lower screen of the two-screen information processing device (transmitting terminal) 100 is the monitoring screen, and the device outputs a sound corresponding to the monitoring screen.

[0176] Furthermore, in the opposite configuration to that shown in Figure 35, the system may be configured to output audio corresponding to the image displayed on the upper screen through the speaker. For example, as shown in Figure 36, the two-screen information processing device (transmitting terminal) 100 outputs sound corresponding to the image displayed on the upper screen. In Figure 36, the image display example d1 (D1) shows that the image displayed on the upper screen of the two-screen information processing device (transmitting terminal) 100 is the monitoring screen, so it outputs a sound corresponding to the monitoring screen. On the other hand, in the image display example d2 of Figure 36 (D2), the image displayed on the upper screen of the two-screen information processing device (transmitting terminal) 100 is the game execution screen, so the game execution sound is output.

[0177] Furthermore, with reference to Figure 37, another example of the use of the two-screen information processing device (transmitting terminal) 100 will be explained.

[0178] Figure 37, "(A) Image displayed on the terminal," shows an example where the game execution screen of a single game application is displayed using the display area of ​​both screens of the dual-screen information processing device (transmitting terminal) 100. This game app is, for example, an app provided by the management server (game app distribution server) 50, and is a game app composed of images with an aspect ratio of 21:18.

[0179] The information processing device (transmission terminal) 100 receives a game application composed of an image with an aspect ratio of 21:18 from a management server (game application distribution server) 50, and displays it in the display areas of both screens of the information processing device (transmission terminal) 100 to execute the game.

[0180] Furthermore, the information processing device (transmission terminal) 100 also receives, as a game application image for live distribution, a game application for single-screen smartphone display with an aspect ratio of, for example, 21:9.

[0181] The information processing device (transmission terminal) 100 receives these two application image data, uses one for display on its own terminal, and uses the other for live distribution. Regarding the image specifications (aspect ratio, resolution, frame rate) of the live distribution image, they can be selected from various specifications as described, for example, by referring to FIG. 11 above.

[0182] However, in the process described by referring to FIG. 37, the information processing device (transmission terminal) 100 needs to receive application image data with two different aspect ratios from the management server (game application distribution server) 50. Without receiving application image data with two different aspect ratios in this way, it is also possible to receive only one game application composed of an image with an aspect ratio of 21:18 corresponding to two screens from the management server (game application distribution server) 50, and perform a process of generating and distributing distribution images with different aspect ratios.

[0183] Referring to FIG. 38, a specific example of the process when performing this process will be described. The self-terminal display image in FIG. 38(A) shows an example of displaying a game execution screen of one game application using the display areas of both screens of the information processing device (transmission terminal) 100 with a two-screen configuration. This game application is, for example, an application provided by the management server (game application distribution server) 50 and is a game application composed of an image with an aspect ratio of 21:18.

[0184] The streaming user 20 selects an arbitrary area from the game execution screen displayed on both screens of the dual-screen information processing device (transmitting terminal) 100. For example, they might crop the area indicated by the dotted line in the diagram. This cropping process extracts an image for distribution, such as an image with a 21:9 aspect ratio, and distributes it.

[0185] The user operation for cropping is performed, for example, by tracking a frame that roughly defines the cropping area (the dotted line frame shown in the diagram) with their finger, or by tapping the center of the image area that they want to distribute. In response to these user operations, the information processing device generates and distributes a distribution image by performing image cropping according to a pre-set aspect ratio of the distribution image.

[0186] [6. Regarding the audio output control processing of the information processing device] Next, we will explain the audio output control process of the information processing device.

[0187] Referring to Figure 39, an example of voice output control performed by the information processing device (transmitting terminal) 100 of this disclosure will be described.

[0188] The display unit of the information processing device (transmitting terminal) 100 shown in Figure 39 displays the game execution screen of the game being played by the distribution user 20.

[0189] The streaming user 20 is listening to the game audio of a game running on the information processing device (transmitting terminal) 100 via the headset 201. The Headset 201 is equipped with a microphone and speaker.

[0190] The headset 201 is connected to the analog audio input / output section of the information processing device (transmitting terminal) 100, such as a 3.5mm audio jack. The information processing device (transmitting terminal) 100 outputs analog audio data of the game sound to the headset 201, and the streaming user 20 can hear the game sound through the speaker of the headset 201. Furthermore, the audio from the streaming user 20, such as game commentary audio, is input to the information processing device (transmitting terminal) 100 via the analog audio input / output section of the headset 201's microphone.

[0191] Furthermore, the digital data output unit of the information processing device (transmitting terminal) 100, such as a Type-C USB output unit, is connected to the PC 203 via the capture card 202. The input of the capture card 202 is, for example, an HDMI® input unit.

[0192] In other words, from the digital data output unit (capture card connection unit) of the information processing device (transmission terminal) 100, (a) Game image (b) Game audio data (c) Game commentary audio from streaming users These data are output and input to PC203 via capture card 202.

[0193] Based on this input data, the PC203 can perform game distribution processing, or record data including game screens, audio, and commentary sounds.

[0194] Thus, the information processing device (transmitting terminal) 100 of this disclosure is For example, it has a configuration that allows parallel execution of audio input / output processing via an analog audio input / output section such as a 3.5mm audio jack, and digital data output processing including audio data via a digital data output section such as a Type-C USB.

[0195] With this configuration, the distribution user 20 can output the image, voice, and the voice of the distribution user 20 of the game to the PC 203 while listening to the voice of the game app via the headset 201, and perform live distribution of the game via the PC 203, game recording processing, etc.

[0196] Referring to FIG. 40 and below, an operation example in the case of executing such voice output setting processing will be described.

[0197] First, as shown in FIG. 40 (step S11), a main menu (UI) 101 is displayed on the display unit of the information processing apparatus 100.

[0198] As described above with reference to FIGS. 3 to 4, the main menu (UI) 101 is displayed by performing a flick process of sliding a finger from the upper part to the lower part of the screen of the information processing apparatus 100, for example.

[0199] Next, in (step S12), the record & stream icon (Rec&stream) displayed on the main menu 101 is selected and operated (tapped).

[0200] Next, as shown in (step S13) at the lower right of FIG. 41, the external output setting menu (UI) selection unit 220 at the upper left end is operated (tapped). By this icon selection process, the external output setting menu (UI) 104 is displayed.

[0201] As shown in FIG. 41, the external output setting menu (UI) 104 includes (a) Image input / output path setting unit 221 (b) Voice input / output path setting unit 222 These two types of setting units are displayed.

[0202] Here, the processing using the “(b) Voice input / output path setting unit 222” will be described. Furthermore, by scrolling the screen of the information processing device 100, the distribution user 20 can sequentially display various setting items in "(b) Audio input / output path setting unit 222". A specific example of the audio input / output path setting process using "(b) Audio Input / Output Path Setting Unit 222" will be explained with reference to Figures 42 and 43.

[0203] First, when the streaming user 20 scrolls through the "(b) Audio Input / Output Path Setting Unit 222" screen shown in Figure 41, the setting items shown in Figure 42 are displayed. The following settings are displayed on the screen of "(b) Audio Input / Output Path Setting Unit 222" shown in Figure 42. (b1) Audio output setting unit 231 to external device (b2) Microphone output setting unit 232 for external devices

[0204] The "(b1) Output audio external device setting unit 231" is a setting unit for selecting whether or not to perform audio output to an external device such as the headset 201 or PC 203, as explained with reference to Figure 39. The configuration offers three options: DP, 3.5mm, and UA. In the example shown in the diagram, DP and 3.5mm are selected.

[0205] DP stands for DisplayPort, and in the example shown in Figure 39, it is a digital output port connected to the capture card 202. 3.5mm is a 3.5mm jack, and in the example shown in Figure 39, it is the analog input / output section connected to the headset 201. UA is the USB audio output section.

[0206] In Figure 42, the setting state of "(b1) Audio output setting unit 231 to external device" is such that both DP and 3.5mm are selected. This setting enables audio output via the two audio output units, as explained earlier with reference to Figure 39. In other words, it becomes possible to perform audio input / output processing via analog audio input / output sections such as a 3.5mm audio jack and digital data output processing including audio data via digital data output sections such as a Type-C USB in parallel.

[0207] The "(b2) Output mic external device setting unit 232" is a setting unit that sets whether or not to output microphone audio to an external device such as the PC203 as explained with reference to Figure 39, and further allows selection of the microphone input (source).

[0208] The microphone input (source) options are 3.5mm and UA, and you can individually configure whether or not to output from each of these microphone inputs (sources). 3.5mm is a 3.5mm jack, and in the example shown in Figure 39, it is the analog input / output section connected to the headset 201. UA is the USB audio output section.

[0209] In the example shown in Figure 42, 3.5 mm is selected. With this setting, as explained earlier with reference to Figure 39, the information processing device 100 can output microphone audio input from the headset 201 to an external device such as the PC 203.

[0210] By further scrolling the screen of "(b) Audio Input / Output Path Setting Unit 222" shown in Figure 42, the setting screen shown in Figure 43 will be displayed.

[0211] The screen of "(b) Audio Input / Output Path Setting Unit 222" shown in Figure 43 displays a volume setting unit for adjusting the volume balance of each audio output to external devices such as the PC203 shown in Figure 39. Specifically, it is the following setting unit. (b3) Microphone output volume setting unit 233 for external devices (b4) Media output volume setting section 234 (b5) Receiver volume setting unit 235

[0212] For example, in the configuration described with reference to Figure 39, the audio output from the information processing device 100 to the PC 203 is a mix of multiple audio sources. Therefore, if the volume of one audio source is too high, problems arise such as the other audio sources becoming inaudible.

[0213] To solve this problem, the volume of each audio (each sound source) is made adjustable, allowing for adjustment of the volume balance of each audio. The UI shown in Figure 43 is a UI for this volume adjustment process for each audio (each sound source), and the output volume of each sound source can be adjusted individually.

[0214] Furthermore, the output audio setting process in the information processing device 100 disclosed herein can be performed not only using the UI launched from the game application's startup screen, but also when the game application is not running.

[0215] Specifically, by performing processes such as those shown in Figures 44 and 45, it is possible to display and configure a UI for setting the output audio to an external device.

[0216] The specific processing sequence is as follows: (Step S51) From the Settings menu, select Device Connection. (Step S52) From the Device Connection menu, select Connection Preference. (Step S53) From the Connection Preference menu, select Audio. (Step S54) In the Audio menu, execute the settings for each audio.

[0217] The audio menu shown in Figure 45 (step S54) is a UI that allows for audio settings similar to the capture card compatible menu 104 explained earlier with reference to Figures 42 and 43.

[0218] Thus, the output audio setting process in the information processing device 100 of this disclosure can be performed not only using the UI launched from the game application's startup screen, but also when the game application is not running.

[0219] As described above, the information processing device 100 of this disclosure can adjust the output volume for each sound source and output it to an external device such as a PC. A specific example of processing that applies this output voice control processing will be explained with reference to Figure 46 and subsequent figures.

[0220] Figure 46 shows the information processing device 100, as well as the headset 201 and PC 203, which are external devices as previously explained with reference to Figure 39.

[0221] In the configuration shown in Figure 46, the streaming user 20 runs a game application on PC 203 and also uses an information processing device 100, such as a smartphone, to make a call (video call) using a calling application.

[0222] Game sounds are input from PC203 to information processing unit 100. Furthermore, the voice of the streaming user 20, who is running a calling application using information processing unit 100, is input to information processing unit 100 from the microphone of the headset 201.

[0223] In this state, the information processing device 100 receives the following two different sounds, namely: (a) Input sound from PC203 (game sound), (b) The incoming audio of the call application, which is the voice of the caller received by the call application running on the information processing device 100, These two different audio data streams are output to the headset 201 worn by the streaming user 20.

[0224] When performing this type of processing, by adjusting the output volume of each sound source using the external output setting menu (UI) explained earlier with reference to Figure 43, it becomes possible to output audio data with the optimal balance from the information processing device 100 to the headset 201.

[0225] The "Information Processing Device Output Audio Adjustment Process" shown in the upper right of Figure 46 illustrates an example of this output audio adjustment process. The "(a) Information Processing Device Input" shown in the diagram represents the volume of two sound sources as pitch. The upper level corresponds to the volume of game sound input from PC203. The lower level corresponds to the volume of the received audio from the call application.

[0226] In the information processing device 100, by adjusting the output volume of each sound source using the external output setting menu (UI) explained earlier with reference to Figure 43, etc., it is possible to generate audio data with an optimal balance, as shown in "(b) Information output device output" in the figure, and this data can be output from the information processing device 100 to the headset 201.

[0227] Furthermore, as shown in Figure 47, the information processing device 100 can also control the audio data output to the PC 203.

[0228] The "(2) Output audio adjustment processing of the information processing device" shown in the lower left of Figure 47 is an example of this output audio adjustment processing. The audio data output from the information processing device 100 to the PC 203 is: (1) Game sound (2) The voice received by the calling app, (3) Microphone input audio It consists of these three types of audio data.

[0229] In the lower left of Figure 47, "(a) Information Processing Device Input" within "(2) Information Processing Device Output Audio Adjustment Processing" indicates the volume of the three sound sources mentioned above as pitch. The top level corresponds to the game sound, the middle level to the received audio from the calling application, and the bottom level to the microphone input audio volume.

[0230] In the information processing device 100, by adjusting the output volume of each sound source using the external output setting menu (UI) explained earlier with reference to Figure 43, etc., it is possible to generate audio data with an optimal balance, as shown in "(b) Information output device output" within "(2) Information processing device output audio adjustment processing" shown in the lower left of Figure 47. This data can then be output from the information processing device 100 to the PC 203, and output from the PC 203's speaker or recorded in the PC 203's memory unit.

[0231] Furthermore, another specific example of processing to which output voice control processing performed in the information processing device 100 of this disclosure is applied will be described with reference to Figure 48 and subsequent figures. Figure 48 shows an example of conducting a music session at two remote locations: a live music venue 310 and the participant's home 330.

[0232] At the live venue 310, guitars 1 and 2 are playing. Meanwhile, at home 330, user 331 is singing along to the guitar playing. Information processing devices a and 311 are located at the live venue 310, and information processing devices b and 312 are located at the home 330. These information processing devices are capable of voice control processing similar to the information processing device (transmission terminal) 100 described earlier. In other words, they are devices capable of voice control processing using the UI described with reference to Figures 40 to 45.

[0233] The information processing devices a and 311 at the live venue 310 and the information processing devices b and 312 at the home 330 are each running the music session application provided by the music session service server 320, and are configured to communicate with each other via the music session service server 320.

[0234] Guitar sound 1 from guitar 1 and guitar sound 2 from guitar 2 at live venue 310 are input to information processing devices a and 311, and transmitted from information processing devices a and 311 to information processing devices b and 312 at home 330 via music session service server 320. The guitar sound is output from the information processing device b,312 to the headset worn by user 331, who is singing, allowing user 331 to sing while listening to the guitar sound through the headset.

[0235] Furthermore, the vocal sound V1, which is the song sung by user 321, is input from the headset microphone to information processing devices b and 312, and then transmitted from information processing devices b and 312 to information processing devices a and 311 at the live venue 310 via the music session service server 320. The information processing device a,311 outputs the vocal sound V1 received from the music session service server 320 through a speaker near the guitarist. This process allows the guitarist to play in sync with user 331's vocal track V1.

[0236] In such a music session, for example, the information processing devices b and 312 at home 330 receive guitar sound 1 from guitar 1 and guitar sound 2 from guitar 2 at the live venue 310 via the music session service server 320, and output the input guitar sounds to the headset worn by user 331.

[0237] The information processing device b,312 performs the output audio adjustment process for this guitar sound. The "Information Processing Device Output Audio Adjustment Process" shown in the upper right of Figure 48 illustrates an example of this output audio adjustment process. The "(a) Information Processing Device Input" shown in the diagram indicates the volume of the two sound sources as pitch. The upper part corresponds to the volume of the sound from guitar 1, which is guitar 1 note G1. The lower part corresponds to the volume of the sound from guitar 2, which is guitar 2 note G2.

[0238] In the information processing device b,312, by adjusting the output volume of each sound source using the external output setting menu (UI) explained earlier with reference to Figure 43, etc., it is possible to generate audio data with an optimal balance, as shown in "(b) Information Output Device Output" within the "Information Processing Device Output Audio Adjustment Process" shown in the upper right of Figure 48. This data can then be output from the information processing device b,312 to the headset worn by the user 331.

[0239] Furthermore, as shown in Figure 49, the information processing device b,312 performs volume adjustment processing on three audio data: vocal sound V1, guitar 1 sound G1 (the sound of guitar 1), and guitar 2 sound G2 (the sound of guitar 2). The output audio data generated as a result of this processing can be transmitted to, for example, the music session service server 320. The music session service server 320 can then perform processing such as distributing the received music session data to multiple terminals or recording it in its storage unit.

[0240] The "(2) Output audio adjustment processing of the information processing device" shown in Figure 49 is an example of output audio adjustment processing performed by the information processing device b,312 during the above processing. The audio data output from information processing device b,312 to music session service server 320 is: (1) The sound of guitar 1 is guitar 1 note G1 (2) The sound of guitar 2 is guitar 2 note G2 (3) Vocal sound V1 It consists of these three types of audio data.

[0241] In Figure 49, "(a) Information Processing Device Input" within "(2) Information Processing Device Output Audio Adjustment Processing" shows the volume of the three sound sources as pitch. The top level corresponds to the volume of guitar 1 (G1), the middle level to guitar 2 (G2), and the bottom level to vocal sound (V1).

[0242] In the information processing device b,312, by adjusting the output volume of each sound source using the external output setting menu (UI) explained earlier with reference to Figure 43, etc., it is possible to generate audio data with an optimal balance, as shown in "(b) Information output device output" within "(2) Information processing device output audio adjustment processing" in Figure 49. The information processing device b,312 transmits this data to the music session service server 320.

[0243] The music session service server 320 can perform processes such as distributing music session data received from information processing devices b and 312 to a large number of terminals, or recording it in a storage unit.

[0244] [7. Examples of Information Processing Device Configurations] Next, we will explain an example of the configuration of an information processing device.

[0245] Figure 50 is a block diagram showing the functional configuration of the information processing device 400 of this disclosure, which is an information processing device that is a user terminal such as a smartphone. The information processing device 400 shown in Figure 50 corresponds to the information processing device (transmitting terminal) 100 and information processing device (receiving terminal) 200 described earlier with reference to Figure 1 and others, as well as information processing devices a, 311 and information processing devices b, 312 described with reference to Figure 48 and others.

[0246] As shown in Figure 50, the information processing device 400 includes an operation unit 401, a storage unit 402, an imaging unit 403, a sensor unit 404, a display unit 405, a microphone 406, a speaker 407, a communication unit 408, external device connection units 1~n, 409-1~n, and a control unit 410.

[0247] The control unit 401 detects various user operations, such as device operations on applications. The above-mentioned device operations include, for example, touch operations and the process of connecting external devices to the information processing device 100. Here, "touch operation" refers to various contact actions on the display unit 405, such as tapping, double-tapping, swiping, and pinching. Furthermore, touch operation includes bringing an object, such as a finger, close to the display unit 405. For this purpose, the operation unit 401 includes, for example, a touch panel, buttons, a keyboard, a mouse, a proximity sensor, etc. The operation unit 401 also inputs information related to detected user operations to the control unit 410.

[0248] The memory unit 402 is a memory area for temporarily or permanently storing various programs and data. For example, the memory unit 402 stores programs and data for the information processing device 100 to perform various functions. Specifically, the memory unit 402 stores programs for running various applications, management data for managing various settings, etc.

[0249] For example, the settings information that the user has configured for various menus (UI), such as the data distribution pre-configuration menu (UI) 102, the data distribution in-progress configuration menu (UI) 103, and the external output configuration menu (UI) 104, as explained earlier, is recorded. However, the above is merely an example, and the type of data stored in the memory unit 402 is not particularly limited.

[0250] The imaging unit 403, based on control by the control unit 410, captures images of, for example, the face of a user operating the information processing device 100. The imaging unit 403 is equipped with an image sensor. A smartphone, which is an example of the information processing device 100, is equipped with a front camera on the display unit 405 side for taking pictures of the user's face, etc., and a main camera on the back side of the display unit 405 for taking pictures of scenery, etc. In this embodiment, as an example, the device controls the shooting with the front camera.

[0251] The sensor unit 404 has the function of collecting sensor information related to the user's actions using various sensors. The sensor unit 404 includes, for example, an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, a vibration sensor, and a GNSS (Global Navigation Satellite System) signal receiver.

[0252] The sensor unit 404, for example, uses a gyro sensor to detect when the user holds the information processing device 100 sideways, and inputs the detected information to the control unit 410.

[0253] The display unit 405 displays various visual information based on control by the control unit 410. The display unit 405 may display, for example, images or text related to the application. The display unit 405 may be equipped with various display devices, such as a liquid crystal display (LCD) device or an organic light-emitting diode (OLED) display device.

[0254] Furthermore, under the control of the control unit 410, the display unit 405 overlays various menus (UI) on a layer above the display application screen.

[0255] Microphone 406 is equipped with a microphone that collects voices and other sounds emitted by the user based on control by control unit 410. Speaker 407 outputs various types of audio. For example, it outputs audio and sounds according to the application status based on control by the control unit 410.

[0256] The communication unit 408 functions as a transmitting and receiving unit for data communication via networks such as Wi-Fi, Bluetooth® (BT), the Internet, and local area networks, and communicates with external devices. For example, it performs processes such as distributing various data externally, including game data.

[0257] External device connection sections-1~n,409-1~n are connection sections used for connecting to the headset 201, PC 203, capture card 202, etc., as previously explained with reference to Figure 39, etc. Specifically, they include, for example, analog audio input / output sections such as a 3.5mm audio jack, and digital data output sections such as a Type-C USB output section.

[0258] The control unit 410 controls, for example, the display data output to the display unit 405 and the transmission data transmitted via the communication unit 408. Furthermore, the control unit 410 performs processes to display various menus (UI) on the display unit 405 based on user operations on the operation unit 401, and processes to record the settings of distribution data corresponding to user operations on the displayed menus (UI) in the storage unit 402.

[0259] Furthermore, when the control unit 410 distributes data via the communication unit 408, it determines the specifications of the distribution data (images, audio) according to the setting information recorded in the storage unit 402 and then distributes it.

[0260] The menus displayed on the display unit 405 under the control of the control unit 410 include, for example, the data distribution pre-setting menu (UI) 102, the data distribution setting menu (UI) 103, and the external output setting menu (UI) 104 described earlier. The control unit 410 records the user-defined settings for these various menus (UI) in the storage unit 402. When distributing data via the communication unit 408, the specifications of the distribution data (images, audio) are determined and distributed according to the setting information recorded in the storage unit 402.

[0261] Specific processes controlled by the control unit 110 include, for example, the following processes (a) to (d). (a) The display unit 405 outputs a display image and the transmission image are generated as two different images. For example, two images are generated that differ in at least one of the aspect ratio, resolution, or frame rate, one of the two generated images is displayed on the display unit 405, and the other is transmitted via the communication unit 408.

[0262] (b) When the information processing device 100 is in a horizontal orientation, it generates and transmits a transmission image having a horizontal layout determined according to the user operation, and when the information processing device 100 is in a vertical orientation, it generates and transmits a transmission image having a vertical layout determined according to the user operation.

[0263] (c) Audio data is generated and transmitted with the volume of each sound source adjusted according to the volume of each sound source determined by the user operation on the audio adjustment menu displayed on the display unit 105. If the settings are changed in response to the user operation on the settings menu during data distribution, the volume of each sound source is changed and transmitted according to the new settings.

[0264] (d) Image display control is performed to switch the display data for the display unit 105 from a display image with a predetermined specification according to user operation to a transmitted image sent via the communication unit 408.

[0265] While an example of the functional configuration of the information processing device 400 has been described so far, the above functional configuration explained using Figure 50 is merely an example, and the functional configuration of the information processing device 100 according to this embodiment is not limited to this example. For example, the information processing device 400 does not necessarily have to have all of the configurations shown in Figure 50, and some of the configurations can be provided in a different device from the information processing device 400. The functional configuration of the information processing device 400 according to this embodiment can be flexibly modified according to specifications and operation.

[0266] Alternatively, the functions of each component may be performed by a computing device such as a CPU (Central Processing Unit) reading a control program, which describes the processing procedures for realizing these functions, from a storage medium such as ROM (Read Only Memory) or RAM (Random Access Memory), interpreting that program, and executing it. Therefore, it is possible to change the configuration used as appropriate depending on the technological level at the time of implementing this embodiment.

[0267] [8. Examples of hardware configurations for information processing devices and servers] Next, with reference to Figure 51, we will describe an example of a hardware configuration for a device that can be used as an information processing device or a management server.

[0268] The hardware of the information processing device shown in Figure 51 corresponds to the hardware of the information processing device (transmitting terminal) 100 and the information processing device (receiving terminal) 200, which were previously explained with reference to Figure 1 and others, as well as the hardware of information processing devices a, 311 and information processing devices b, 312, which were explained with reference to Figure 48 and others. It also corresponds to the hardware of the management server 50, which was explained with reference to Figure 1 and others, and the music session service server 320, which was explained with reference to Figure 48 and others. The following describes each component that makes up the hardware of the information processing device shown in Figure 51.

[0269] The CPU (Central Processing Unit) 501 functions as a control unit and data processing unit that execute various processes according to the programs stored in the ROM (Read Only Memory) 502 or the storage unit 508. For example, it executes processes according to the sequence described in the above embodiment. The RAM (Random Access Memory) 503 stores programs and data that the CPU 501 executes. These CPU 501, ROM 502, and RAM 503 are interconnected by a bus 504.

[0270] The CPU 501 is connected to the input / output interface 505 via the bus 504. The input / output interface 505 is connected to an input unit 506 consisting of various switches, a keyboard, a mouse, a microphone, etc., and an output unit 507 that outputs data to a display unit, a speaker, etc. The CPU 501 executes various processes in response to commands input from the input unit 506 and outputs the processing results to, for example, the output unit 507.

[0271] The storage unit 508, connected to the input / output interface 505, consists of, for example, a hard disk and stores programs executed by the CPU 501 and various data. The communication unit 509 functions as a transmitting and receiving unit for data communication via Wi-Fi communication, Bluetooth® (BT) communication, and other networks such as the Internet and local area networks, and communicates with external devices.

[0272] The drive 510 connected to the input / output interface 505 drives removable media 511 such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory such as memory cards, and performs data recording or reading.

[0273] [9. Summary of the structure of this disclosure] The embodiments of this disclosure have been described in detail above with reference to specific examples. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of this disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted restrictively. To determine the gist of this disclosure, one should refer to the claims section.

[0274] Furthermore, the technology disclosed in this specification can have the following configuration. (1) It has a control unit that controls the display data output to the display unit and the transmission data transmitted via the communication unit, The control unit, An information processing device that generates two types of images, one with different specifications for the display image constituting the display data and the other with different specifications for the transmission image constituting the transmission data, displays one of the two generated images on the display unit, and transmits the other via the communication unit.

[0275] (2) The control unit, The information processing apparatus according to (1), which generates two types of images having different aspect ratios, resolutions, or frame rates, displays one of the two generated images on the display unit, and transmits the other via the communication unit.

[0276] (3) The display unit has a configuration having two screens, The control unit, As the display image, an image having an aspect ratio that can be displayed in the entire area of ​​the two-screen display unit is generated. The information processing apparatus according to (2) that generates an image having an aspect ratio that can be displayed on a single-screen display unit as the transmitted image.

[0277] (4) The control unit, A data distribution pre-configuration menu for performing the settings of the transmission data before data transmission is displayed on the display unit. The storage unit stores setting information corresponding to user operations on the pre-configuration menu before data distribution. An information processing device according to any one of (1) to (3), which generates and transmits transmission data according to the setting information stored in the storage unit when data transmission is performed.

[0278] (5) The pre-configuration menu before data distribution is: It has a distribution image editing unit for editing the aforementioned transmitted images, The control unit, The information processing device according to (4), which generates and transmits a transmission image having a layout determined in response to user operations on the aforementioned distribution image editing unit.

[0279] (6) The image editing unit included in the pre-set menu before data distribution is: A horizontal layout when the aforementioned information processing device is oriented horizontally, The aforementioned information processing device is configured to allow editing of two different layouts: a vertical layout when the device is oriented horizontally, and a horizontal layout. The control unit, The information processing device according to (5), which generates and transmits a transmission image having two types of layouts: a horizontal layout and a vertical layout, determined in response to user operations on the aforementioned distribution image editing unit.

[0280] (7) The control unit, If the information processing device is in a horizontal orientation, it generates and transmits a transmission image having a horizontal layout determined according to user operation. The information processing device according to (6), wherein when the information processing device is in a vertical orientation, it generates and transmits a transmission image having a vertical layout determined according to user operation.

[0281] (8) The image editing unit included in the pre-set menu before data distribution is: An information processing apparatus according to any one of (5) to (7), having a configuration that enables editing processing of multiple region units constituting the transmitted image.

[0282] (9) The image editing unit for distribution included in the pre-set menu before data distribution is: An information processing apparatus according to any one of (5) to (8), having a configuration that enables editing processing for each of the text area, image area, and background area that constitute the transmitted image.

[0283] (10) The image editing unit for distribution included in the pre-setting menu before data distribution is: An information processing device as described in any of (5) to (9), which has a configuration that allows the specification and editing of the title included in the transmitted image.

[0284] (11) The pre-configuration menu before data distribution is: It has a thumbnail editing unit for editing the thumbnails that make up the transmitted image, The control unit, An information processing device according to any of (4) to (10), which generates and transmits a thumbnail having a layout determined in response to user operations to the thumbnail editing department.

[0285] (12) The thumbnail editing unit included in the pre-set menu before data distribution is: The information processing apparatus according to (11), having a configuration that enables editing processing for each of the text area, image area, and background area that constitute the thumbnail.

[0286] (13) The pre-configuration menu before data distribution is: It has an audio adjustment menu for adjusting the audio data that constitutes the transmitted data, The aforementioned audio adjustment menu is: The information processing device according to any one of (4) to (12), wherein the volume of each sound source constituting the audio data included in the transmitted data can be adjusted.

[0287] (14) The control unit, The information processing device according to (13), which generates and transmits audio data in which the volume of each sound source is adjusted according to the volume of each sound source determined in response to user operation on the sound adjustment menu.

[0288] (15) The control unit, An information processing apparatus according to any one of (1) to (14) that performs image display control to switch the display data for the display unit from the displayed image to a transmitted image transmitted via the communication unit in response to user operation.

[0289] (16) The control unit, The information processing apparatus according to (15), which receives the transmitted image via the communication unit and displays it on the display unit.

[0290] (17) The control unit, A data distribution settings menu for executing the settings of the transmission data during data transmission is displayed on the display unit. The setting information corresponding to the user operation on the data distribution setting menu is stored in the storage unit. An information processing device according to any one of (1) to (16) that modifies the settings of the transmission data according to the setting information stored in the memory unit and then transmits it.

[0291] (18) The data distribution settings menu mentioned above is: It has an audio adjustment menu for adjusting the audio data that constitutes the transmitted data, The aforementioned audio adjustment menu is: The configuration allows for adjustment of the volume of each sound source that constitutes the audio data included in the transmitted data. The control unit, The information processing device according to (17), which generates and transmits audio data with the volume of each sound source changed according to the volume of each sound source changed in response to user operation on the sound adjustment menu.

[0292] (19) An information processing method to be performed in an information processing device, The information processing device has a control unit that controls the display data output to the display unit and the transmission data transmitted via the communication unit. The control unit, An information processing method comprising generating two types of images, one with different specifications for the display image constituting the display data and the other with different specifications for the transmission image constituting the transmission data, displaying one of the two generated images on the display unit, and transmitting the other via the communication unit.

[0293] (20) A program that causes information processing to be performed in an information processing device, The information processing device has a control unit that controls the display data output to the display unit and the transmission data transmitted via the communication unit. The program controls the control unit, A program that generates two types of images, one with different specifications for the display image constituting the display data and the other with different specifications for the transmission image constituting the transmission data, and performs the process of displaying one of the two generated images on the display unit and transmitting the other via the communication unit.

[0294] Furthermore, the series of processes described in the specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer embedded in dedicated hardware, or the program can be installed and executed on a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing from the recording medium to a computer, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as a built-in hard disk.

[0295] Furthermore, the various processes described in this specification may not only be executed sequentially as described, but may also be executed in parallel or individually as needed, depending on the processing capacity of the device performing the process. In addition, in this specification, a system is a logical combination of multiple devices, and the devices in each configuration are not limited to being located in the same enclosure. [Industrial applicability]

[0296] As described above, according to the configuration of one embodiment of the present disclosure, a device and method are realized in which display data and transmission data are controlled in an information processing device, and the transmission data is generated and transmitted according to pre-configured information generated by the user. Specifically, the system includes a control unit that controls the display data output to the display unit and the transmission data transmitted via the communication unit. The control unit generates two types of images with different specifications: the display image that constitutes the display data and the transmission image that constitutes the transmission data. For example, the aspect ratio, resolution, and frame rate of the transmission data are set in advance, and the transmission data is generated according to these settings and transmitted via the communication unit.

[0297] These processes enable the control of both display data and transmission data in an information processing device, and realize a device and method that generates and transmits transmission data according to pre-configured information generated by the user. [Explanation of symbols]

[0298] 10 Information Processing Systems 50 Management Server 100 Information Processing Device (Transmission Terminal) 101 Main Menu 102 Pre-configuration menu (UI) before data distribution 103 Data Distribution Settings Menu (UI) 104 External Output Settings Menu (UI) 120 Selection section of the distribution pre-configuration menu (UI) 121 Title Editorial Department 122 Display Information Editorial Department 123 Resolution and frame rate settings section 124 Viewing permission target setting section 125 Stream delay setting section 126 Thumbnail editing and display section 127 Streaming image editing and display section 128 Privacy settings screen editing and display section 129 Microphone Settings Section 130 Volume setting section 131 Extension Settings UI Migration Section 181 Audio Adjustment Unit 182 Privacy Settings Screen Adjustment Section 183 UI display setting section 184 Display Image Switching Section 200 Information Processing Device (Receiving Terminal) 201 Headset 202 Capture Card 203 PC 222 Audio Input / Output Path Setting Section 311,312, Information Processing Device 400 Information Processing Devices 401 Operation section 402 Storage section 403 Imaging Unit 404 Sensor Unit 405 Display section 406 Mike 407 Speakers 408 Communications Department 409 External device connection section 410 Control Unit 501 CPU 502 ROM 503 RAM 504 Bus 505 Input / Output Interface 506 Input section 507 Output section 508 Storage section 509 Communications Department 510 Drive 511 Removable Media

Claims

1. A display unit and The system includes a control unit that controls the display data output to the display unit and the transmission data transmitted via the communication unit. The control unit, Two types of images are generated, one with different specifications for the display image constituting the display data and the other with different specifications for the transmission image constituting the transmission data. One of the two generated images is displayed on the display unit, and the other is transmitted via the communication unit. The control unit, A data distribution pre-configuration menu for performing the settings of the transmission data before data transmission is displayed on the display unit. The storage unit stores setting information corresponding to user operations on the pre-configuration menu before data distribution. An information processing device that generates and transmits transmission data according to the setting information stored in the storage unit when data transmission is performed.

2. The control unit, The information processing apparatus according to claim 1, which generates two types of images having different aspect ratios, resolutions, or frame rates, displays one of the two generated images on the display unit, and transmits the other via the communication unit.

3. The aforementioned display unit has a configuration that includes a two-screen display unit. The control unit, As the display image, an image having an aspect ratio that can be displayed in the entire area of ​​the two-screen display unit is generated. The information processing apparatus according to claim 2, which generates an image having an aspect ratio that can be displayed on a single-screen display unit as the transmitted image.

4. The aforementioned data distribution pre-configuration menu is: It has a distribution image editing unit for editing the aforementioned transmitted images, The control unit, The information processing apparatus according to claim 1, which generates and transmits a transmission image having a layout determined in response to user operations on the aforementioned distribution image editing unit.

5. The image editing unit included in the pre-set menu before data distribution is: A horizontal layout when the aforementioned information processing device is oriented horizontally, The aforementioned information processing device is configured to allow editing of two different layouts: a vertical layout when the device is oriented horizontally, and a horizontal layout. The control unit, The information processing apparatus according to claim 4, which generates and transmits a transmission image having two types of layouts: a horizontal layout and a vertical layout, determined in response to user operations on the aforementioned distribution image editing unit.

6. The control unit, If the information processing device is in a horizontal orientation, it generates and transmits a transmission image having a horizontal layout determined according to user operation. The information processing device according to claim 5, wherein, when the information processing device is in a vertical orientation, it generates and transmits a transmission image having a vertical layout determined according to user operation.

7. The image editing unit included in the pre-set menu before data distribution is: The information processing apparatus according to claim 4, having a configuration that enables editing processing of multiple region units constituting the transmitted image.

8. The image editing unit included in the pre-set menu before data distribution is: The information processing apparatus according to claim 4, having a configuration that enables editing processing for each of the text area, image area, and background area that constitute the transmitted image.

9. The image editing unit included in the pre-set menu before data distribution is: The information processing apparatus according to claim 4, which has a configuration that allows specifying and editing the title included in the transmitted image.

10. The aforementioned data distribution pre-configuration menu is: It has a thumbnail editing unit for editing the thumbnails that make up the transmitted image, The control unit, The information processing device according to claim 1, which generates and transmits a thumbnail having a layout determined in response to user operations on the thumbnail editing unit.

11. The thumbnail editing unit included in the pre-configuration menu before data distribution is: The information processing apparatus according to claim 10, having a configuration that enables editing processing for each of the text area, image area, and background area that constitute the thumbnail.

12. The aforementioned data distribution pre-configuration menu is: It has an audio adjustment menu for adjusting the audio data that constitutes the transmitted data, The aforementioned audio adjustment menu is: The information processing apparatus according to claim 1, wherein the configuration allows for adjustment of the volume of each sound source unit that constitutes the audio data included in the transmitted data.

13. The control unit, The information processing device according to claim 12, which generates and transmits audio data with the volume of each sound source adjusted according to the volume of each sound source determined in response to a user operation on the sound adjustment menu.

14. The control unit, The information processing apparatus according to claim 1, which performs image display control to switch the display data for the display unit from the displayed image to a transmitted image transmitted via the communication unit in response to a user operation.

15. The control unit, The information processing apparatus according to claim 14, which receives the transmitted image via the communication unit and displays it on the display unit.

16. The control unit, A data distribution settings menu for executing the settings of the transmission data during data transmission is displayed on the display unit. The setting information corresponding to the user operation on the data distribution setting menu is stored in the storage unit. The information processing apparatus according to claim 1, which modifies the settings of the transmission data according to the setting information stored in the storage unit and then transmits it.

17. The aforementioned data distribution settings menu is: It has an audio adjustment menu for adjusting the audio data that constitutes the transmitted data, The aforementioned audio adjustment menu is: The configuration allows for adjustment of the volume of each sound source that constitutes the audio data included in the transmitted data. The control unit, The information processing apparatus according to claim 16, which generates and transmits audio data with the volume of each sound source changed according to the volume of each sound source changed in response to a user operation on the sound adjustment menu.

18. This is an information processing method executed in an information processing device. The aforementioned information processing device is Display unit and The system includes a control unit that controls the display data output to the display unit and the transmission data transmitted via the communication unit. The control unit, Two types of images are generated, one with different specifications for the display image constituting the display data and the other with different specifications for the transmission image constituting the transmission data. One of the two generated images is displayed on the display unit, and the other is transmitted via the communication unit. The control unit, A data distribution pre-configuration menu for performing the settings of the transmission data before data transmission is displayed on the display unit. The storage unit stores setting information corresponding to user operations on the pre-configuration menu before data distribution. An information processing method that generates and transmits transmission data according to the setting information stored in the memory unit when data transmission is performed.

19. This is a program that executes information processing in an information processing device. The aforementioned information processing device is Display unit and The system includes a control unit that controls the display data output to the display unit and the transmission data transmitted via the communication unit. The program controls the control unit, The process involves generating two types of images, one with different specifications for the display image constituting the display data and the other with different specifications for the transmission image constituting the transmission data; displaying one of the two generated images on the display unit; and transmitting the other via the communication unit. The control unit, A process to display a data distribution pre-configuration menu on the display unit for performing the settings of the transmission data before data transmission, A process to store setting information in the storage unit in response to user operations on the pre-set menu before data distribution, A program that, when data transmission is performed, generates and transmits transmission data according to the setting information stored in the memory unit.

Images