Information processing system and information processing method

Abstract

There is room for improvement in how the processing unit is utilized when it is capable of generating image data and performing processing for multiple applications. [Solution] When the first processing unit executes the first application processing and the second application processing, the second processing unit executes a process to generate second image data with a second resolution lower than the first resolution based on the first application processing in a second period shorter than the first period within the unit period, and executes a process to generate third image data based on the second application processing in a third period different from the second period within the unit period, and the output unit outputs an output image to the display that includes a second region smaller than the first region showing the second image data and a third region different from the second region showing the third image data.

Description

Technical Field

[0005] ,

[0001] The present disclosure relates to an information processing system and an information processing method.

Background Art

[0002] An information processing system that performs an image generation process for generating an image by an application and a process different from the image generation process is known (for example, Japanese Patent Application Laid-Open No. 2018-010688).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a processing unit capable of generating image data executes processes related to a plurality of applications, there is room for improvement in how to use the processing unit.

Means for Solving the Problems

[0005] (Configuration 1) An information processing system according to one embodiment includes a first processing unit capable of executing a first application processing and a second application processing, a second processing unit capable of generating one or more image data for each unit period, and an output unit that outputs an output image based on one or more image data for each unit period to a display. When the first processing unit executes the first application processing and does not execute the second application processing, the second processing unit executes a process to generate first image data of a first resolution based on the first application processing in the first period within the unit period, and the output unit outputs an output image including a first region showing the first image data to a display. When the first processing unit executes the first application processing and the second application processing, the second processing unit executes a process to generate second image data with a second resolution lower than the first resolution based on the first application processing in a second period shorter than the first period within the unit period, and executes a process to generate third image data based on the second application processing in a third period different from the second period within the unit period, and the output unit outputs an output image to the display that includes a second region smaller than the first region representing the second image data and a third region different from the second region that represents the third image data.

[0006] (Configuration 2) In Configuration 1, the second processing unit may perform a predetermined process different from the process of generating image data in a fourth period within the unit period, which is different from the second and third periods.

[0007] (Configuration 3) In Configuration 1 or 2, the information processing system may further include an acquisition unit that acquires imaging data of the real space captured by a camera while at least the first condition is met. The second processing unit performs predetermined processing on the imaging data when the first processing unit performs the first application processing and the second application processing and the first and second conditions are met. When the first condition is met, the fourth period is available regardless of whether the second condition is met or not.

[0008] (Configuration 4) In any of Configurations 1 to 3, the second processing unit may generate third image data based on the second application processing, as well as on at least a portion of the imaging data and the result of performing a predetermined process on the imaging data.

[0009] (Configuration 5) In any of Configurations 1 to 4, the information processing system may further include a communication unit that communicates with other information processing systems. When the first processing unit executes the first application processing and the second application processing, the communication unit may obtain from the other information processing system at least a portion of the image data captured by the other information processing system's camera, and the results of performing predetermined processing on said image data. The second processing unit may generate third image data based on the second application processing and at least a portion of the image data obtained from the other information processing system, and the results of performing predetermined processing.

[0010] (Configuration 6) In any of Configurations 1 to 5, even if the first processing unit executes the first application processing and the second application processing, the second processing unit may generate image data selected from the first image data and the second image data according to user operation, if the first condition is not met.

[0011] (Configuration 7) In any of Configurations 1 to 6, the second processing unit may include at least a first core for executing the process of generating image data and a second core for executing a predetermined process. A switching period may be provided between the process of generating image data and the predetermined process.

[0012] (Configuration 8) In any of Configurations 1 to 7, when the second processing unit performs a predetermined process in the fourth period, it may generate image data with a third resolution lower than the second resolution as second image data based on the first application processing.

[0013] (Configuration 9) In any of Configurations 1 to 8, the information processing system may be able to receive power from an external power source. When the first processing unit executes the first application processing and the second application processing, the resolution of the second image data generated when power is supplied to the information processing system from an external power source may be higher than the resolution of the second image data generated when power is not supplied to the information processing system from an external power source.

[0014] (Configuration 10) In any of Configurations 1 to 9, if the second processing unit fails to complete the generation of the third image data in the third period within a unit period when the first processing unit is executing the first application processing and the second application processing, the second processing unit may continue the process of generating the third image data in the third period within a subsequent unit period.

[0015] (Configuration 11) In any of Configurations 1 to 10, if the image data based on the first application processing includes at least a portion of the image data captured by the camera in real space, a portion of the period during which the process for generating the image data based on the first application processing is performed may be used as a period for performing a predetermined process on the image data.

[0016] (Configuration 12) According to one embodiment, an information processing method is provided in an information processing system including a first processing unit capable of executing a first application processing and a second application processing, and a second processing unit capable of generating one or more image data for each unit period. The information processing method includes the step that, when the first processing unit executes the first application processing and does not execute the second application processing, the second processing unit executes a process to generate first image data of a first resolution based on the first application processing in a first period within the unit period, and outputs an output image including a first region showing the first image data to a display for each unit period. The information processing method includes the steps of: when the first processing unit executes the first application processing and the second application processing, the second processing unit executes a process to generate second image data with a second resolution lower than the first resolution based on the first application processing in a second period shorter than the first period within a unit period; executes a process to generate third image data based on the second application processing in a third period different from the second period within a unit period; and for each unit period, output an output image to a display that includes a second region smaller than the first region representing the second image data and a third region different from the second region that represents the third image data. [Brief explanation of the drawing]

[0017] [Figure 1] This is a schematic diagram showing an example of the configuration of an overall system including an information processing system according to this embodiment. [Figure 2] This is a schematic diagram showing an example of the hardware configuration of a game device according to this embodiment. [Figure 3] This is a schematic diagram showing an example of a screen image of a game device according to this embodiment. [Figure 4] This is a schematic diagram showing an example of a screen image of a game device according to this embodiment. [Figure 5] This is a schematic diagram showing an example of a screen image of a game device according to this embodiment. [Figure 6]It is a schematic diagram showing an example of a display mode of a camera image displayed on a game device according to the present embodiment. [Figure 7] It is a diagram for explaining an example of display processing in a game device according to the present embodiment. [Figure 8] It is a diagram for explaining an example of GPU scheduling in a game device according to the present embodiment. [Figure 9] It is a flowchart showing an example of processing related to a scheduler in a game device according to the present embodiment.

Embodiments for Carrying Out the Invention

[0018] The present embodiment will be described in detail with reference to the drawings. For the same or corresponding parts in the drawings, the same reference numerals are given and their descriptions will not be repeated.

[0019] [A. Operation Example] First, an operation example of an information processing system according to the present embodiment will be described.

[0020] FIG. 1 is a schematic diagram showing a configuration example of an overall system 1 including an information processing system according to the present embodiment. As an example of the information processing system, the game device 100 will be described. However, the information processing system may be, for example, a personal computer, a smartphone, a tablet, a smart TV, or the like.

[0021] The term "information processing system" includes at least both a configuration in which necessary components are implemented in a single housing and a configuration in which necessary components are separately implemented in a plurality of housings.

[0022] Referring to FIG. 1, the overall system 1 includes one or more game devices 100. FIG. 1 shows an overall system 1 including four game devices 100 as an example. The overall system 1 may further include a relay server 300 and a management server 350.

[0023] In the overall system 1, communication is possible between users using the game device 100 via the network 4. More specifically, each game device 100 can exchange communication data 10 for communication with one or more other game devices 100. That is, a game device 100 can send communication data 10 it has generated to one or more other game devices 100, and can also receive communication data 10 from one or more other game devices 100. A game device 100 may perform processing based on the communication data 10 it receives from one or more other game devices 100 (for example, displaying images on the display 120 and outputting sound).

[0024] In the following explanation, a game device 100 that transmits communication data 10 will also be referred to as a "sender". Each game device 100 can be a sender and can also receive communication data 10 from other senders. A certain game device 100 can also receive communication data 10 from multiple other game devices 100. Therefore, from the perspective of that particular game device 100, there will be multiple senders. However, there may be game devices 100 that only receive communication data 10, or game devices 100 that only transmit communication data 10.

[0025] The communication data 10 exchanged between game devices 100 may include at least one of image data or sound data. The communication data 10 may also include mask information or attribute information, as described later.

[0026] The image data may include, for example, image data based on a game application running on the game device 100, or camera image data generated from imaging data captured by a camera 118 available to the game device 100. The image data may also include at least one of the following: moving image data or one or more still image data.

[0027] The sound data may include at least one of the following: sound data based on a game application running on the game device 100, or sound data collected by a microphone available to the game device 100.

[0028] Image data and sound data may be transmitted together or separately. Communication data 10 may be transmitted as stream data.

[0029] The entire mechanism and functionality for communication between users using the game device 100, as described above, will also be referred to as "chat" below. Such chats are intended to involve multiple users. One or more users (or accounts) participating in the same chat may be collectively referred to as a "chat group." Participation in a chat group may be managed on an account-by-account basis for each user.

[0030] The exchange of communication data 10 takes place between game devices 100, but participation in chat groups may be done on a user-by-user or account-by-account basis. One or more accounts may be registered to a single game device 100. In this case, the user using the game device 100 may select an account each time. Alternatively, the user using the game device 100 may operate the game device 100 to access a management server 350 or the like and perform login or other processing, thereby dynamically associating an account with the game device 100.

[0031] The method for exchanging communication data 10 between game devices 100 may be a method that goes through one or more relay entities (e.g., relay servers 300), or it may be a method that transmits directly between game devices 100 (e.g., P2P method).

[0032] The relay server 300 relays the communication data 10 transmitted by the game device 100. More specifically, when the relay server 300 receives communication data 10 from a game device 100, it forwards the received communication data 10 to one or more other game devices 100.

[0033] The management server 350 is responsible for the management necessary for one or more game devices 100 to exchange communication data 10. For example, the management server 350 may create a new chat session and determine whether to allow a game device 100 or a user to join the new chat.

[0034] The relay server 300 and the management server 350 may be implemented on a single physical server. System 1 may include multiple relay servers 300 and multiple management servers 350. The relay servers 300 and the management servers 350 do not need to have separate functions. The relay servers 300 and the management servers 350 may be virtual servers.

[0035] [B. Example of hardware configuration for game device 100] Next, an example of the hardware configuration of the game device 100 according to this embodiment will be described.

[0036] Figure 2 is a schematic diagram showing an example of the hardware configuration of a game device 100 according to this embodiment. The game device 100 is a type of computer.

[0037] Referring to Figure 2, the game device 100 includes, for example, a control circuit 102, an input unit 110, a communication unit 112, a display controller 114, a camera interface 116, a display 120, a sound interface 122, a microphone 124, a speaker 126, a storage 130, and a battery 50.

[0038] The control circuit 102 is responsible for processing performed by the game device 100. The control circuit 102 includes, for example, a CPU (Central Processing Unit) 104, a GPU (Graphics Processing Unit) 106, and memory 108. The control circuit 102 may also be a SoC (System on Chip). However, the CPU 104, GPU 106, and memory 108 may be implemented independently of each other, rather than being implemented in a single control circuit 102.

[0039] The control circuit 102 may include multiple CPUs 104 or multiple GPUs 106. The CPUs 104 may have multiple cores. The GPUs 106 may have multiple cores. The control circuit 102 may include multiple memory locations 108. The memory locations 108 may include areas accessible from both the CPUs 104 and GPUs 106, or areas accessible from only one of the CPUs 104 or GPUs 106. The memory locations 108 are volatile storage media such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory).

[0040] The input unit 110 accepts user input. The input unit 110 includes, for example, a keyboard, mouse, game controller, buttons, a directional pad, etc. The input unit 110 may also be an interface with a device that accepts user input (for example, a game controller). That is, the device that accepts user input may be located outside the game device 100.

[0041] The communication unit 112 exchanges data with other game devices 100, relay servers 300, management servers 350, etc., via the network 4. The communication unit 112 may be connected to the network 4 by wire or by wireless connection.

[0042] The display controller 114 generates at least one of the following based on the image data generated by the control circuit 102: a video signal for displaying an image on the display 120, or a video signal for displaying an image on an external display.

[0043] Display 120 is, for example, an LCD (Liquid Crystal Display) or an OLED display.

[0044] Part or all of the display controller 114 may be included in the control circuit 102.

[0045] The camera interface 116 receives imaging data from the camera 118. The camera interface 116 may be connected to the camera 118 by a wire or by wireless connection.

[0046] The camera 118 generates imaging data (moving image data or still image data) of the real space in accordance with commands from the game device 100 or according to predetermined conditions. The camera 118 includes an image sensor such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. A lens may be placed in front of the image sensor.

[0047] Camera 118 may be, for example, a camera for video calls. Camera 118 may be a dedicated camera that can only be connected to a specific device, or it may be a general-purpose camera that can be connected to any device. Multiple cameras 118 may be connected to the game device 100. Camera 118 may be incorporated into the game device 100. That is, the game device 100 may include one or more cameras.

[0048] In the communication unit 112 and the camera interface 116, wired connections such as USB (Universal Serial Bus) or parallel connections may be used. For wireless connections, Bluetooth®, ZigBee®, or wireless LAN (IEEE 802.11 standard) may be used.

[0049] The sound interface 122 exchanges sound signals with the connected device. The sound interface 122 may include a phone jack. For example, the sound interface 122 acquires a sound signal based on the sound collected by the microphone 124. The sound interface 122 outputs a sound signal based on the sound data generated by the control circuit 102 to the speaker 126.

[0050] The microphone 124 is located in the casing of the game device 100 and collects sound to generate a sound signal. The speaker 126 is located in the casing of the game device 100 and generates sound from the sound signal. At least one of the microphone 124 or speaker 126 may be located outside the game device 100. For example, headphones or a headset may be connected to the phone jack of the sound interface 122.

[0051] The storage 130 is, for example, a non-volatile storage medium such as flash memory and a hard disk. The storage 130 stores, for example, a system program 132, a chat application 134, and one or more game applications 136.

[0052] The system program 132 includes computer-readable instructions for basic processing such as hardware control of the game device 100 and providing a program execution environment. The system program 132 may also be an OS (Operating System).

[0053] The chat application 134 includes computer-readable instructions necessary to implement the chat described above. Part or all of the chat application 134 may be included in the system program 132.

[0054] The game application 136 includes computer-readable instructions for running the game.

[0055] The storage 130 may include, for example, a storage medium that can be attached to the game device 100, such as a cartridge and an optical disc. In this case, the game device 100 may include an interface for connecting to the storage medium.

[0056] In this specification, the term "processor" includes, at a minimum, the CPU 104 and GPU 106, and hardwired circuits such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field Programmable Gate Arrays).

[0057] In this specification, the term “memory” includes at least the memory 108 and the storage 130.

[0058] The battery 50 supplies power to various parts of the game device 100. The battery 50 is electrically connected to circuits for charging and discharging, and to circuits for supplying power (neither of which are shown in the diagram).

[0059] [C. Screen image example] Next, an example of a screen image displayed by the game device 100 will be described.

[0060] The game device 100 may provide a kind of voice chat function by running a chat application 134. Voice chat is achieved, for example, by the exchange of sound data between the game devices 100.

[0061] Furthermore, the game device 100 may provide a kind of image distribution function by running a chat application 134. Image distribution is achieved, for example, by exchanging image data between game devices 100. Image distribution can also be described as a kind of video chat. In image distribution, image data showing at least a portion of the screen image displayed on the display 120 of the sending game device 100 may be sent to other game devices 100.

[0062] Furthermore, in image distribution, camera image data generated from imaging data captured by a camera 118 available to the source game device 100 may be transmitted to other game devices 100. The user operating the game device 100 may turn the function for transmitting camera image data on or off.

[0063] Hereinafter, the function by which game device 100 transmits camera image data to other game devices 100 will also be referred to as "use of camera images." "Use of camera images is on" means that the processing of image data captured by at least the camera 118 of game device 100 is enabled. When the condition that use of camera images is on is met, the control circuit 102 or camera interface 116 acquires image data of the real space captured by the camera 118. Only when use of camera images is on may the processing of displaying camera image data acquired from other game devices 100 be enabled.

[0064] In the following explanation, "Voice chat is on" means that at least one of the following is enabled: the ability to send sound data to other users in the chat group, or the ability to receive sound data from other users in the chat group.

[0065] In the following description, "video streaming off" means at least one of the following: either no users in the chat group are streaming video, or the game device 100 is set not to display other users' video streams. "video streaming on" means at least one of the following: a user using the game device 100 is streaming video, or the game device 100 is set to display other users' video streams.

[0066] In addition to voice chat or video streaming, text chat may also be possible. When text chat is performed, the communication data 10 may include text information transmitted by each game device 100. Images showing such text may also be displayed.

[0067] Figures 3 to 5 are schematic diagrams showing example screen images of a game device 100 according to this embodiment.

[0068] Figure 3 shows an example of a screen image 128 when the game application 136 is running. In Figure 3, the chat application 134 is not running. Therefore, both voice chat and video streaming are turned off.

[0069] In the state shown in Figure 3, the screen image 128 displayed on the display 120 of the game device 100 includes a game image 152 based on the game application 136. The game image 152 spans the entire display area of ​​the display 120. In this example, the screen image 128 matches the game image 152. The game device 100 generates game image data (game image data 152D, described later) that corresponds to the entire display area of ​​the display 120.

[0070] A user using the game device 100 plays the game while viewing the game image 152 as a screen image 128.

[0071] Figures 4 and 5 show examples of screen images when the chat application 134 is running. In Figures 4 and 5, it is assumed that in addition to the chat application 134, the game application 136 is also running.

[0072] Figure 4 shows an example of a screen image when voice chat is on and video streaming is off. The screen image 128 displayed on the display 120 of the game device 100 includes a main image 150M and an icon image 160.

[0073] A game image 152M based on the game application 136 is displayed as the main image 150M. The main image 150M may, for example, occupy about 4 / 5 of the total display area of ​​the display 120.

[0074] Icon image 160 indicates an account participating in the chat group. The display of icon image 160 may change in accordance with the corresponding user's speech. For example, the icon image 160 of a user who is currently speaking may change in size, color, and brightness in accordance with their speech.

[0075] Users using the game device 100 can play the game while viewing game images 152M, and also enjoy conversations with other users participating in the chat group. The game application 136 may run a competitive game that one or more other users can also participate in. Users who are available to participate in or currently participating in the competitive game do not necessarily have to be the same as users participating in the chat group.

[0076] Figure 5 shows an example of a screen image when voice chat and video streaming are enabled, and camera image usage is also enabled. In this case, the communication data 10 includes image data showing at least a portion of the screen image and camera image data. Users using the game device 100 can share the screen image of the game they are playing with other users.

[0077] Referring to Figure 5, the screen image 128 displayed on the display 120 of the game device 100 includes the main image 150S, the distribution display images 140-1 to 140-4 (hereinafter sometimes collectively referred to as "distribution display images 140"), and the icon image 160.

[0078] The main image 150S is a game image 152S based on the game application 136. The main image 150S may, for example, occupy about two-thirds of the total display area of ​​the display 120.

[0079] The streaming display images 140-2 to 140-4 are generated based on communication data 10 received from other game devices 100 associated with other accounts participating in the chat group. The streaming display image 140-1 may be generated based on communication data 10 that game device 100 sends to other game devices 100 associated with other accounts participating in the chat group. In other words, the streaming display image 140-1 may be for monitoring the communication data 10 sent by the device itself.

[0080] However, the distribution display image 140-1 may also be generated based on communication data 10 received from other game devices 100 associated with other accounts participating in the chat group.

[0081] Each of the streaming display images 140-1 to 140-4 includes a game image 142, a user icon 144, a username 146, and a camera image 148.

[0082] Game image 142 corresponds to game image 152 based on game application 136 running on the source game device 100. Game image 142 is not limited to game image 152 based on game application 136 running on the source game device 100, but may be any image generated by the source game device 100.

[0083] User icon 144 indicates the account selected on the sending game device 100. User icon 144 may be pre-configured or generated on the sending game device 100. Username 146 indicates the user information corresponding to the account on the sending game device 100.

[0084] The camera image 148 is based on the image capture data from the camera 118 available to the transmitting game device 100. The user may be able to choose whether or not to allow the transmission of camera images on the transmitting game device 100. The management server 350 may manage the permission or denial of camera image transmission for each game device 100 or for each account.

[0085] At least one of the user icon 144 or username 146 may not be displayed. Also, at least one of the game image 142 or camera image 148 may not be displayed. In addition to these images, or in place of either of these images, another image may be displayed.

[0086] As shown in Figure 4, the screen image 128 is displayed on the display 120, allowing the user of the game device 100 to play the game while viewing the game status (or game image 142) of other users.

[0087] As shown in Figures 4 and 5, when a user plays a game while communicating with other users, the game application 136 and the chat application 134 are executed in parallel on the game device 100.

[0088] Game image 152M shown in Figure 4 and game image 152S shown in Figure 5 are images based on the game application 136, similar to game image 152 shown in Figure 3. However, the resolution of game image 152M is lower than that of game image 152 shown in Figure 3. The resolution of game image 152S is lower than that of game image 152M. The subscripts "M" and "S" are added to clarify these differences in resolution. Therefore, in contexts where resolution is not a focus, they are also collectively referred to as "game image 152".

[0089] In this specification, “resolution” primarily means screen resolution or number of pixels. “Resolution” may be expressed as the product of the number of pixels on each of the two sides (for example, 1920 pixels × 1080 pixels) or as the total number of pixels in the image (for example, 2,073,600 pixels).

[0090] The size of the area occupied by the main images 150M, 150S and the distribution display image 140 may be variable.

[0091] [D. Example of camera image display mode] Next, we will describe an example of the display mode of the camera image 148 included in the streamed display image 140.

[0092] Figure 6 is a schematic diagram showing an example of a display mode for a camera image 148 displayed in a game device 100 according to this embodiment. In Figure 6, the area showing the game image 142 is indicated by hatching to clarify the display mode of the camera image 148.

[0093] As shown in Figures 6(A) to 6(D), in the streamed display image 140, the camera image 148 may be superimposed on the game image 142.

[0094] Figure 6(A) shows an example of a camera image 148A in face display mode. In face display mode, an image is displayed in which a person's face has been cut out from the image captured by camera 118. The transmitting game device 100 detects the region of a person's face from the image captured by camera 118, extracts the outline of the detected person's face, and generates mask information showing the extracted outline.

[0095] Figure 6(B) shows an example of a camera image 148A in person display mode. In person display mode, an image is displayed which is a cropped version of the image captured by camera 118, showing the area of ​​the person. The transmitting game device 100 extracts the contour of the person from the image captured by camera 118 and generates mask information showing the extracted contour.

[0096] Figure 6(B) shows an example where the background of the person is set to white, but any image may be set as the background. In this case, image processing such as overlaying the person's image on the background image may be performed.

[0097] Figure 6(C) shows an example of a camera image 148C in unprocessed mode. In unprocessed mode, the image shown by the camera 118's imaging data is displayed as is. The game device 100 transmits the camera 118's imaging data directly to other game devices 100 as camera image data corresponding to the camera image 148C.

[0098] Figure 6(D) shows an example of a camera image 148D in background blur mode. In background blur mode, an image is displayed in which areas other than the person are blurred from the image data captured by camera 118. The transmitting game device 100 identifies the area of ​​the person in the image shown by the image data captured by camera 118 and blurs the area other than the identified person.

[0099] The transmitting game device 100 generates camera image data corresponding to the display mode as described above. The camera image data includes at least a portion of the image data captured by the camera 118 capturing the real space.

[0100] The display mode of the camera image 148 as described above may be set by the transmitting game device 100, or it may be determined by the receiving game device 100.

[0101] [E. Example of display processing in game device 100] Next, an example of display processing in the game device 100 will be described.

[0102] Figure 7 is a diagram illustrating an example of display processing in a game device 100 according to this embodiment.

[0103] Referring to Figure 7, the CPU 104 is capable of executing the game application 136 and the chat application 134. The CPU 104 executes the processing of the game application 136 according to the computer-readable instructions contained in the game application 136. The CPU 104 executes the processing of the chat application 134 according to the computer-readable instructions contained in the chat application 134.

[0104] The CPU 104 issues a drawing command 1360 to the GPU 106 by executing the game application 136. The drawing command 1360 includes, for example, instructions for rendering game image data 152D according to the game progress of the game application 136. For example, the drawing command 1360 may be issued periodically according to the frame rate of the display 120. The drawing command 1360 may also include a specification of the resolution of the game image 152. The GPU 106 generates the game image data 152D according to the drawing command 1360.

[0105] The CPU 104 issues a drawing command 1340 to the GPU 106 by executing the chat application 134. The drawing command 1340 includes instructions for generating an image based on the chat application 134. The drawing command 1340 includes, for example, instructions for rendering the distribution display image data 140D. The GPU 106 generates the distribution display image data 140D according to the drawing command 1340, using game image data 142D, camera image data 174D, and mask information 176, etc., which are included in the communication data 10 acquired by the communication unit 112.

[0106] If the distribution display image 140-1 shown in Figure 5 above is based on communication data 10 to be transmitted to another game device 100, the GPU 106 generates distribution display image data 140D corresponding to the distribution display image 140-1 based on at least a portion of the image data from the camera 118 and mask information generated from said image data.

[0107] On the other hand, the distribution display images 140-2 to 140-4 shown in Figure 5 above are based on communication data 10 acquired from other game devices 100. Therefore, the GPU 106 generates distribution display image data 140D corresponding to the distribution display images 140-2 to 140-4 based on camera image data 174D and mask information 176 acquired from other game devices 100. By acquiring camera image data 174D and mask information 176 from other game devices 100, the game device 100 does not need to perform processing such as generating mask information for other game devices 100. In other words, although the predetermined processing on the image data is relatively computationally intensive, each game device 100 transmits the results of the predetermined processing performed on the image data captured by the camera 118 available in its own device to the other game devices 100, so the predetermined processing is not executed redundantly.

[0108] The transmitting game device 100 sends the camera image data 174D and mask information 176 to another game device 100, which can then perform processes such as setting an arbitrary image as the background for areas other than a person's face, or areas other than a person. Alternatively, the areas other than a person's face, or areas other than a person, may be made transparent.

[0109] If the chat application 134 is running and camera image usage is enabled, the CPU 104 may issue an image processing instruction 1342 to the GPU 106. The image processing instruction 1342 includes, for example, an instruction for generating camera image data 170D using the image data captured by the camera 118.

[0110] The instructions included in the image processing instruction 1342 are changed according to the mode shown in Figures 6(A) to 6(D). For example, when the face display mode shown in Figure 6(A) or the person display mode shown in Figure 6(B) is selected, the image processing instruction 1342 includes instructions for generating the region of a person's face or mask information 172 indicating the region of a person. The image processing instruction 1342 may also specify, for example, a process that uses deep learning. More specifically, the image processing instruction 1342 may include instructions for detecting the region of a person's face from the image data captured by the camera 118 using deep learning.

[0111] GPU106 may include circuitry that enables use as a GPGPU (General-Purpose computing on Graphics Processing Units), such as for deep learning. For example, GPU106 may include a cluster that includes a core for executing processes that follow drawing instructions 1340 (e.g., rendering) and a core for executing processes that follow image processing instructions 1342 (e.g., processes using deep learning). For example, if only one core can be enabled at any given time in the cluster, the core is switched after issuing drawing instructions 1340.

[0112] The GPU 106 generates camera image data 170D, or camera image data 170D and mask information 172, based on the image data captured by the camera 118 acquired via the camera interface 116, in accordance with the image processing instruction 1342.

[0113] The drawing instruction 1340 or 1360 issued by the CPU 104 may include instructions for generating icon image data 160D and overlay image data 158D. For example, the icon image data 160D represents the icon image 160 shown in Figure 4 or Figure 5. The overlay image data 158D includes an image representing an icon image or message that is overlaid on the display 120. The GPU 106 may generate at least one of the icon image data 160D and the overlay image data 158D according to the drawing instruction 1340 or 1360.

[0114] GPU106 periodically generates one or more image data (e.g., game image data 152D, streaming display image data 140D, icon image data 160D, and overlay image data 158D) for each unit period.

[0115] The display controller 114 outputs an output image to the display 120 based on one or more image data (for example, game image data 152D, distribution display image data 140D, icon image data 160D, and overlay image data 158D) for each unit period. At least a portion of the processing handled by the display controller 114 may be executed by the GPU 106.

[0116] For example, if the frame rate is 60 [fps (frames per second)], the unit period may be 1 / 60 [sec], which corresponds to the time of one frame. The unit period may be changed depending on the state of the game device 100 and the application being executed. The frame rate may also change according to the refresh rate of the display 120.

[0117] The communication unit 112 transmits communication data 10, which includes game images, camera image data 170D, or mask information 172. The resolution of the game image 152 included in the communication data 10 may be lower than the resolution of the game image data 152D generated by the GPU 106.

[0118] Instead of directly generating the streaming display image data 140D, the GPU 106 may generate game image data 142D, user icon 144, username 146, and camera image 148, respectively, and the display controller 114 may superimpose these images to generate the streaming display image 140 or output image.

[0119] The "image data" shown in Figure 7 can also be referred to as "rendering data." Each image data may be temporarily stored in an arbitrary intermediate buffer. The intermediate buffer may be located in any area of ​​memory 108, for example, or in a part of the display controller 114.

[0120] GPU106 processes multiple instructions issued by CPU104. Scheduler180 manages the time available for processing each instruction on GPU106. Scheduler180 may, for example, be part of the functionality provided by system program132. Scheduler180 may also manage execution on GPU106 on a process or thread basis. The following describes an example of managing computing resources on GPU106.

[0121] [F. Example of managing computing resources in game device 100] Next, we will describe an example of managing computing resources in the game device 100.

[0122] As shown in Figure 3, if the game application 136 is running and the chat application 134 is not running, the GPU 106 of the game device 100 only needs to perform the process of generating game image data 152D. That is, the GPU 106 performs the process of generating game image data 152D based on the game application 136 at each unit period. The resolution of the game image data 152D matches, for example, the resolution of the entire display area of ​​the display 120. The display controller 114 outputs an output image containing the game image 152 showing the game image data 152D to the display 120.

[0123] As shown in Figure 4, when the game application 136 and the chat application 134 are running and voice chat is enabled, the display 120 displays the main image 150M in addition to the icon image 160. The GPU 106 performs the process of generating game image data 152D based on the game application 136 for each unit period, and also performs the process of generating icon image data 160D for displaying the icon image 160.

[0124] The main image 150M shown in Figure 4 is smaller than the entire display area of ​​the display 120. Therefore, the resolution of the game image data 152D is lower than the resolution of the game image data 152D representing game image 152. The display controller 114 outputs an output image to the display 120 that includes the game image 152M representing the game image data 152D and the icon image 160 representing the icon image data 160D.

[0125] As shown in Figure 5, when the game application 136 and the chat application 134 are running and voice chat and image streaming are enabled, the display 120 displays the streaming display image 140 and the icon image 160 in addition to the main image 150S. The GPU 106 performs the process of generating game image data 152D based on the game application 136 for each unit period, and also performs the process of generating streaming display image data 140D for displaying the streaming display image 140 and icon image data 160D for displaying the icon image 160.

[0126] Furthermore, if camera image usage is enabled, the game device 100 generates camera image data 170D (or, depending on the mode, camera image data 170D and mask information 172). The GPU 106 executes the process of generating camera image data 170D (or camera image data 170D and mask information 172) at regular intervals. The period at which camera image data 170D (or camera image data 170D and mask information 172) is generated may differ from the period at which image data is generated.

[0127] The main image 150S shown in Figure 5 is smaller than the entire display area of ​​the display 120. Therefore, the resolution of the game image data 152D is lower than the resolution of the game image data 152D that shows game image 152. The display controller 114 outputs an output image to the display 120 that includes the game image 152S that shows game image data 152D, the distribution display image 140 that shows distribution display image data 140D, and the icon image 160 that shows icon image data 160D.

[0128] The resolution of game image 152M shown in Figure 4 and game image 152S shown in Figure 5 is lower than the resolution of game image 152 shown in Figure 3. In other words, the processing load required to generate game image data 152D in the state of Figure 4 or Figure 5 is expected to be lower than the processing load required to generate game image data 152D in the state of Figure 3. To put it another way, the computing resources required to display game images 152M and 152S are less than the computing resources required to display game image 152.

[0129] In the game device 100, the allocation of computing resources provided by the GPU 106 may be changed depending on whether or not the chat application 134 is running.

[0130] Figure 8 is a diagram illustrating an example of scheduling for the GPU 106 in a game device 100 according to this embodiment. The GPU schedule shown in Figure 8 may be set by the scheduler 180 for the GPU 106.

[0131] For example, if drawing is performed according to a drawing command 1340 issued in response to the execution of the chat application 134, and then drawing is performed according to a drawing command 1360 issued in response to the execution of the game application 136, a context switch occurs between the two processes. The context switch period shown in Figure 8 indicates the period during which such a context switch occurs.

[0132] The GPU schedule 200 shown in Figure 8(A) corresponds to a state where the game application 136 is running and the chat application 134 is not running (see, for example, Figure 3).

[0133] GPU schedule 200 includes a first resource allocation period 202, a context switch period 204, and a second resource allocation period 206.

[0134] For example, the length of the first resource allocation period 202 may be 5% of the unit period, and the length of the second resource allocation period 206 may be 93% of the unit period. The context switch period 204, which is the time required to switch the execution of the process between the first resource allocation period 202 and the second resource allocation period 206, may be 2% of the unit period. During the first resource allocation period 202, GPU 106 executes the process of generating overlay image data 158D as needed.

[0135] During the second resource allocation period 206, the GPU 106 executes a process to generate game image data 152D based on the game application 136, which corresponds to the game image 152. For example, during the second resource allocation period 206, game image data 152D with a resolution corresponding to the entire display area of ​​the display 120 is generated.

[0136] The GPU schedule 210 shown in Figure 8(B) corresponds to a state where image streaming is turned off while the game application 136 and the chat application 134 are running (see, for example, Figure 4). Since image streaming is turned off, the use of camera images is also turned off.

[0137] The GPU schedule 210 includes a first resource allocation period 212, a context switch period 214, and a second resource allocation period 216.

[0138] For example, the length of the first resource allocation period 212 may be 10% of the unit period, and the length of the second resource allocation period 216 may be 88% of the unit period. The context switch period 214 may be 2% of the unit period.

[0139] During the first resource allocation period 212, the GPU 106 executes the process of generating icon image data 160D corresponding to the icon image 160. The GPU 106 may also execute the process of generating overlay image data 158D as needed.

[0140] During the second resource allocation period 216, the GPU 106 executes the process of generating game image data 152D based on the game application 136, which corresponds to game image 152M or game image 152S. The second resource allocation period 216 is shorter than the second resource allocation period 206 shown in Figure 8(A).

[0141] The GPU schedule 220 shown in Figure 8(C) corresponds to a state where video streaming and camera image usage are enabled while the game application 136 and the chat application 134 are running (see, for example, Figure 5).

[0142] GPU schedule 220 includes a first resource allocation period 222, a first context switch period 224, a third resource allocation period 226, a second context switch period 228, and a second resource allocation period 230.

[0143] For example, the length of the first resource allocation period 222 may be 10% of the unit period, the length of the third resource allocation period 226 may be 10% of the unit period, and the length of the second resource allocation period 230 may be 76% of the unit period. The first context switch period 224 and the second context switch period 228 may both be 2% of the unit period.

[0144] During the first resource allocation period 222, the GPU 106 executes the process of generating distribution display image data 140D corresponding to the distribution display image 140, and also executes the process of generating icon image data 160D corresponding to the icon image 160. The GPU 106 may also generate overlay image data 158D as needed. For example, since it is necessary to generate the distribution display image data 140D, the first resource allocation period 222 is longer than the first resource allocation period 212 shown in Figure 8(B).

[0145] During the third resource allocation period 226, the GPU 106 performs a predetermined process different from the process of generating image data (or rendering). For example, the GPU 106 generates camera image data 170D, or camera image data 170D and mask information 172, based on the image data captured by the camera 118. More specifically, the GPU 106 performs processes on the image data such as detecting the face region of a person, detecting a person, extracting the contours of the detected faces, and generating mask information 172 that shows the extracted contours. As a result of performing the predetermined processes on the image data, for example, mask information 172 is generated. In this way, the computing resources of the GPU 106 may be allocated so that the GPU 106 can be used as a GPGPU.

[0146] The first context switch period 224 is a period provided for switching between the process of generating image data and a predetermined process.

[0147] During the second resource allocation period 230, the GPU 106 executes the process of generating game image data 152D based on the game application 136, which corresponds to the game image 152S. For example, since the resolution of the game image 152S shown in Figure 5 is lower than the resolution of the game image 152M shown in Figure 4, the second resource allocation period 230 is set to be shorter than the second resource allocation period 216 shown in Figure 8(B).

[0148] If camera image utilization is enabled, predetermined processing is performed on the image data captured by camera 118, requiring the allocation of computing resources for this predetermined processing. On the other hand, since the area occupied by the game image 152S on display 120 is relatively small, the computing resources allocated to generating the game image data 152D corresponding to the game image 152S can be reduced, allowing the necessary processing to be performed within the computing resources of GPU 106.

[0149] Thus, when the GPU 106 performs a predetermined process during the third resource allocation period 226, it performs a process to generate game image data 152D with an even lower resolution as game image data based on the game application 136.

[0150] The camera image 148 may have several display modes (see Figure 6) selectable. Of the display modes described above, in face display mode (see Figure 6(A)) or person display mode (see Figure 6(B)), the process of generating mask information 172 is executed. On the other hand, in unprocessed mode (see Figure 6(C)) or background blur mode (see Figure 6(D)), the process of generating mask information 172 is unnecessary.

[0151] Therefore, the predetermined processing described above is executed when the conditions that camera image utilization is turned on and that a specific display mode for the camera image 148 is selected are met. Consequently, the amount of computing resources required by the GPU 106 may change depending on the display mode of the camera image 148.

[0152] However, the same GPU schedule may be applied regardless of the display mode of the camera image 148. That is, when the condition that camera image utilization is turned on is met, the third resource allocation period 226 may be kept available regardless of whether the condition that a specific display mode for the camera image 148 is selected is met.

[0153] The GPU schedule described above indicates the allocation of computing resources, and processing is not guaranteed to occur for the entire allocated period. In other words, there may be periods of idleness during the allocated time.

[0154] The above GPU schedule is just one example, and the length of the allocation period may be changed as needed. The following describes some variations of the GPU schedule.

[0155] In one embodiment, when the game application 136 and the chat application 134 are running, and video streaming is turned on but camera image usage is turned off, either GPU schedule 210 or GPU schedule 220 may be selected. Alternatively, a different GPU schedule may be provided for the case where video streaming is turned on but camera image usage is turned off.

[0156] In one embodiment, when the game application 136 and the chat application 134 are running, and video streaming is turned on but the use of camera images is turned off, the predetermined processing of the image data captured by the camera 118 is not performed. Therefore, since it is not necessary to allocate the computing resources of the GPU 106 to the predetermined processing, the screen image shown in Figure 3 may be displayed on the display 120 in response to user operation, rather than the screen image shown in Figure 4 or Figure 5. That is, the GPU 106 may generate game image data 152D corresponding to the game image 152 (see Figure 3). In this case, the GPU schedule 200 or the GPU schedule 210 may be appropriately selected in response to user operation. If the game image data 152D corresponding to the game image 152 can be generated during the second resource allocation period 216, the GPU schedule 210 may be maintained.

[0157] In one embodiment, the processing power of the GPU 106 can be increased when the game device 100 is powered by an external power source. For example, the game device 100 may be connectable to a cradle. The cradle may include an interface for supplying power to the game device 100 from an external power source, and an interface for connecting the game device 100 to an external display.

[0158] When the game device 100 operates using power supplied from the battery 50, the clock speed of the GPU 106 may be reduced to suppress power consumption. On the other hand, when power is supplied from an external power source, the processing power of the GPU 106 can be utilized to its fullest extent. As a result, even with the same allocation period, the GPU 106 can perform more processing. If the processing load on the image data is approximately constant, the third resource allocation period 226 for performing the predetermined processing can be shortened. As a result, the second resource allocation period 216 can also be lengthened. By increasing the processing power of the GPU 106, or by lengthening the second resource allocation period 216, at least one of these can generate higher-resolution game image data 152D even when video streaming and camera image utilization are enabled. Thus, the resolution of the game image data 152D generated when power is supplied from an external power source may be higher than the resolution of the game image data 152D generated when power is not supplied from an external power source.

[0159] In one embodiment, during the first resource allocation period 222, the GPU 106 executes a process to generate distribution display image data 140D corresponding to the distribution display image 140. Since the distribution display image 140 includes one or more image objects as well as a camera image 148, computing resources may become insufficient depending on the circumstances. In such cases, the period for generating or updating the distribution display image data 140D may be lengthened. For example, the GPU 106 generates game image data 152D in accordance with the frame rate of the display 120 (e.g., 60 [fps]). On the other hand, if the GPU 106 cannot generate the distribution display image data 140D in accordance with the frame rate of the display 120, it may generate the distribution display image data 140D at, for example, half the frame rate (e.g., 30 [fps]).

[0160] In this case, if the GPU 106 is unable to complete the generation of the distribution display image data 140D in the first resource allocation period 222 within a unit period corresponding to the length of one frame, for example, it may continue the process of generating the distribution display image data 140D in the first resource allocation period 222 within the unit period of the structure. Since the GPU 106 generates the distribution display image data 140D and the game image data 152D in different resource allocation periods, it is possible to lower only the frame rate of the distribution display image data 140D without affecting the frame rate of the game image data 152D.

[0161] In one embodiment, camera images may be used in the game application 136. In such cases, a portion of the second resource allocation period, which is allocated to execute the process of generating game image data 152D based on the game application 136, may be used as a period for executing image processing related to the camera images. As a result, the second resource allocation period becomes shorter, while the third resource allocation period becomes longer. The computing resources required to generate game image data 152D based on the game application 136 are secured by the second resource allocation period and the third resource allocation period.

[0162] As a variation, in cases where the game image 152 is displayed across the entire display area of ​​the display 120 (see, for example, Figure 3), image processing related to the camera image may be performed during a portion of the second resource allocation period allocated for executing the process of generating game image data 152D based on the game application 136. In other words, the second resource allocation period may be used for the process of generating game image data 152D and for image processing related to the camera image.

[0163] [G. Example of scheduler processing in game device 100] The GPU schedule described above is set or modified by the scheduler 180 provided by the system program 132. An example of the scheduler 180's operations is described below.

[0164] Figure 9 is a flowchart showing an example of processing related to the scheduler 180 in a game device 100 according to this embodiment. The processing procedure shown in Figure 9 may also be realized by the CPU 104 of the game device 100 executing the system program 132.

[0165] Referring to Figure 9, when the CPU 104 is powered on or wakes from sleep (step S2), it sets the GPU schedule 200 (see Figure 8(A)) (step S4). Note that the GPU schedule 200 may be maintained regardless of whether the game application 136 is running or not.

[0166] Next, the CPU 104 determines whether or not the display of the chat menu has been instructed (step S6). The chat menu includes, for example, image objects for the user to use the functions of the chat application 134, as shown in Figure 4 or Figure 5. If the display of the chat menu is not instructed (NO in step S6), the process in step S6 is repeated.

[0167] When instructed to display the chat menu (YES in step S6), the CPU 104 sets the GPU schedule 210 (see Figure 8(B)) (step S8) and executes the chat application 134 (step S10). If the chat application 134 is already running, the process in step S10 may be skipped.

[0168] The CPU 104 determines whether or not the use of the camera image is enabled (step S12).

[0169] If the use of camera images is turned off (NO in step S12), the GPU schedule 210 (see Figure 8(B)) is set (step S14). If the GPU schedule 210 has already been set, the process in step S16 may be skipped.

[0170] On the other hand, if the use of camera images is enabled (YES in step S12), the CPU 104 sets the GPU schedule 220 (see Figure 8(C)) (step S16).

[0171] Next, it is determined whether or not an instruction has been given to end the display of the chat menu (step S18). An instruction to end the display of the chat menu includes at least one of the following: an instruction to end the execution of the chat application 134, or an instruction to switch to a display mode in which the game image 152 is displayed across the entire display area of ​​the display 120 while the chat application 134 remains running.

[0172] If the display of the chat menu is not instructed to end (NO in step S18), the process from step S12 onwards is repeated.

[0173] When instructed to stop displaying the chat menu (YES in step S18), CPU 104 terminates execution of the chat application 134 (step S20) and sets the GPU schedule 200 (see Figure 8(A)) (step S22). Then, the process from step S6 onwards is repeated.

[0174] If a display mode switch is instructed, the process in step S20 may be skipped. In other words, the chat application 134 may remain running.

[0175] [H. Variant] In the above embodiment, if the use of camera images is enabled, the transmitting game device 100 transmits the camera image data 174D and mask information 176 to another game device 100. The other game device 100 can use the camera image data 174D and mask information 176 to perform processes such as setting an arbitrary image as the background. As a variation, the transmitting game device 100 may set an arbitrary image as the background of the camera image data 174D and then transmit it to the other game device 100. In other words, the process of setting or changing the background of the camera image may be performed by the transmitting game device 100.

[0176] Camera image data 174D and mask information 176 acquired from another game device 100 may be used not only as part of the distributed display image 140, but also as part of the game image. For example, the face of a user using the other game device 100, acquired from the other game device 100, may be superimposed on the game image.

[0177] [I. Advantages] According to this embodiment, the GPU schedule can be switched according to the execution status of the application program and the communication program, so that the computing resources of GPU 106 can be used efficiently. When the application program and the communication program are running, the resolution of the image based on the application program may be lower compared to when the communication program is not running, so that more computing resources can be allocated to the processing required by the communication program.

[0178] According to this embodiment, when the use of camera images is enabled, computing resources are allocated to enable the GPU 106 to be used as a GPGPU. This allows image data to be generated using the GPU 106, and advanced image processing can also be performed.

[0179] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims rather than by the foregoing description, and all modifications within the meaning and scope equivalent to the claims are intended to be included. [Explanation of Symbols]

[0180] 1 Overall System, 4 Network, 10 Communication Data, 50 Battery, 100 Game Device, 102 Control Circuit, 104 CPU, 106 GPU, 108 Memory, 110 Input Unit, 112 Communication Unit, 114 Display Controller, 116 Camera Interface, 118 Camera, 120 Display, 122 Sound Interface, 124 Microphone, 126 Speaker, 128 Screen Image, 130 Storage, 132 System Program, 134 Chat Application, 136 Game Application, 140 Streaming Display Image, 140D Streaming Display Image Data, 142, 152, 152M, 152S Game Image, 142D, 152D Game Image Data, 144 User Icon, 146 Username, 148, 148A, 148C, 148D Camera Image, 150M, 150S Main Image, 158D Overlay image data, 160 Icon image, 160D Icon image data, 170D, 174D Camera image data, 172, 176 Mask information, 180 Scheduler, 200, 210, 220 GPU schedule, 202, 212, 222 First resource allocation period, 204, 214 Context switch period, 206, 216, 230 Second resource allocation period, 224 First context switch period, 226 Third resource allocation period, 228 Second context switch period, 300 Relay server, 350 Management server, 1340, 1360 Drawing command, 1342 Image processing command.

Claims

1. A first processing unit capable of executing first application processing and second application processing, A second processing unit capable of generating one or more image data for each unit period, The system includes an output unit that outputs an output image based on the one or more image data to a display for each unit period, If the first processing unit executes the first application processing and does not execute the second application processing, The second processing unit executes a process to generate first image data of a first resolution based on the first application processing during the first period within the unit period. The output unit outputs the output image, which includes a first region showing the first image data, to the display. When the first processing unit executes the first application processing and the second application processing, The second processing unit is, Within the aforementioned unit period, in a second period shorter than the first period, a process is executed to generate a second image data with a second resolution lower than the first resolution, based on the first application processing. In a third period within the aforementioned unit period, which is different from the second period, a process is executed to generate a third image data based on the second application processing. An information processing system that outputs the output image to the display, wherein the output unit includes a second region smaller than the first region representing the second image data and a third region different from the second region and representing the third image data.

2. The information processing system according to claim 1, wherein the second processing unit performs a predetermined process different from the process of generating image data in a fourth period within the unit period, which is different from the second and third periods.

3. The system further includes an acquisition unit that acquires imaging data of the real space captured by a camera, while at least the first condition is met. When the first processing unit executes the first application processing and the second application processing, if the first and second conditions are met, the second processing unit executes the predetermined processing on the imaging data. The information processing system according to claim 2, wherein when the first condition is met, the fourth period is available regardless of whether the second condition is met or not.

4. The information processing system according to claim 3, wherein the second processing unit generates the third image data based on the second application processing and on at least a portion of the imaging data and the result of performing the predetermined processing on the imaging data.

5. It further includes a communication unit that communicates with other information processing systems, When the first processing unit executes the first application processing and the second application processing, the communication unit obtains from the other information processing system at least a portion of the image data captured by the other information processing system's camera of the real space, and the results of performing a predetermined process on said image data. The information processing system according to claim 3, wherein the second processing unit generates the third image data based on the second application processing, at least a portion of the imaging data obtained from the other information processing system, and the result of performing the predetermined processing.

6. The information processing system according to claim 5, wherein the second processing unit generates image data selected from the first image data and the second image data according to user operation, even when the first processing unit executes the first application processing and the second application processing, if the first condition is not met.

7. The second processing unit includes at least a first core for executing a process for generating image data and a second core for executing the predetermined process, The information processing system according to claim 2, wherein a switching period is provided between the process of generating the image data and the predetermined process.

8. The information processing system according to any one of claims 3 to 7, wherein the second processing unit, when executing the predetermined processing in the fourth period, generates image data with a third resolution lower than the second resolution as the second image data based on the first application processing.

9. The aforementioned information processing system can be powered by an external power source. The information processing system according to any one of claims 1 to 7, wherein the second processing unit, when the first processing unit executes the first application processing and the second application processing, has a resolution higher than the resolution of the second image data generated when power is supplied to the information processing system from an external power source.

10. The information processing system according to any one of claims 1 to 7, wherein the second processing unit, when the first processing unit executes the first application processing and the second application processing, continues the process of generating the third image data in the third period of a subsequent unit period if it is unable to complete the generation of the third image data in the third period of the unit period.

11. The information processing system according to any one of claims 1 to 7, wherein if the image data based on the first application processing includes at least a portion of imaging data captured by a camera in real space, a portion of the period during which the process for generating the image data based on the first application processing is performed is used as a period for performing a predetermined process on the imaging data.

12. An information processing method in an information processing system including a first processing unit capable of executing first application processing and second application processing, and a second processing unit capable of generating one or more image data for each unit period, If the first processing unit executes the first application processing and does not execute the second application processing, The second processing unit executes a process to generate first image data of a first resolution based on the first application processing during the first period within the unit period. The steps include outputting an output image to a display for each unit period, which includes a first region showing the first image data, When the first processing unit executes the first application processing and the second application processing, The second processing unit, Within the aforementioned unit period, in a second period shorter than the first period, a process is executed to generate a second image data with a second resolution lower than the first resolution, based on the first application processing. In a third period within the aforementioned unit period, which is different from the second period, a process is executed to generate a third image data based on the second application processing. An information processing method comprising the step of outputting the output image to the display for each unit period, the output image including a second region smaller than the first region showing the second image data and a third region different from the second region showing the third image data.

Images