Information processing systems, information processing methods, and programs

Abstract

This system provides an information processing system that appropriately executes application processing and system processing. [Solution] In the information processing system 1, the game machine 10 includes an application execution unit 13 that executes an application program, an image generation unit 14 that generates images, and an image output unit 15 that outputs the images generated by the image generation unit 13 to a liquid crystal screen 17. The image generation unit 14 sequentially executes system processing and application processing, including rendering processing for applications based on the execution results by the application execution unit 13, within a unit period. If the second processing is not completed within the unit period, the application processing is interrupted, and the system processing and the continuation of the interrupted application processing are sequentially executed within the next unit period. Based on the completion of the system processing and application processing, the image output unit 15 outputs the image to the liquid crystal screen without waiting for the unit period to expire.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and a program for executing an application.

Background Art

[0002] Conventionally, an information processing apparatus including a processing unit that performs execution of system software and communication processing with a server, and also performs processing for generating an image by an application has been known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the information processing apparatus as described above, there has been room for improvement in the scheduling of application processing and system processing by the processing unit. In view of the above background, an object of the present invention is to provide an information processing system that can appropriately execute application processing and processing different from the application processing.

Means for Solving the Problems

[0005] (Configuration 1) The information processing system of Configuration 1 comprises an application execution unit that executes an application program, an image generation unit that generates an image, and an image output unit that outputs the image generated by the image generation unit to a display screen. The image generation unit sequentially executes a first process based on processing data at a first timing and a second process including rendering for the application based on the execution result by the application execution unit within a first unit period. If the second process is not completed within the first unit period, the second process is interrupted. Within the second unit period following the first unit period, the first process based on processing data at a second timing later than the first timing and the continuation of the interrupted second process are executed sequentially. The image output unit outputs the image to the display screen without waiting for the unit period to expire, based on the completion of at least the second process.

[0006] (Configuration 2) In the information processing system of Configuration 1, the processing data is updated according to the execution result of a system process different from the application program, the first process includes a system rendering process according to the result of the system process, and the image output unit may output the image to the display screen without waiting for the expiration of the unit period, based on the completion of the system rendering process and the second process.

[0007] (Composition 3) The information processing system of configuration 2 is capable of communicating with other information processing devices and includes an information processing device equipped with the display screen, further comprising: an image generation unit that performs the first processing including transmission processing; and a communication unit that transmits information based on the results of the transmission processing to the other information processing device, wherein the image based on the results of the transmission processing may not be output to the display screen of the information processing system but may be output to the display screen of the other information processing device.

[0008] (Composition 4) In any of the information processing systems of configurations 1 to 3, the image generation unit may execute the first process before the second process within the unit period.

[0009] (Composition 5) In the information processing system of configuration 4, the second process may be configured not to be executed until a predetermined time has elapsed since the first process was executed.

[0010] (Composition 6) In the information processing system of configuration 4, the first process is completed when either the processing completion condition is met or a predetermined time has elapsed since the first process was executed. The second process may be executed without waiting for the predetermined time to elapse if the first process is completed based on the fulfillment of the processing completion condition.

[0011] (Composition 7) In any of the information processing systems of configurations 1 to 6, the length of the unit period may be determined according to the application program executed by the application execution unit.

[0012] (Composition 8) Any of the information processing systems in configurations 1 to 7 includes an information processing device and a display device, wherein the information processing device comprises the application execution unit, the image generation unit and the image output unit, and the display device may comprise the display screen.

[0013] (Composition 9) The information processing method of configuration 9 is an information processing method in an information processing system comprising an application execution unit that executes an application program, an image generation unit that generates an image, and an image output unit that outputs the image generated by the image generation unit to a display screen, comprising the steps of: the image generation unit sequentially executing a first process based on processing data at a first timing and a second process including rendering for an application based on the execution result by the application execution unit within a first unit period; if the second process is not completed within the first unit period, interrupting the second process and sequentially executing a first process based on processing data at a second timing later than the first timing and the continuation of the interrupted second process within the second unit period following the first unit period; and the image output unit outputting the image to the display screen without waiting for the unit period to expire, based on the completion of at least the second process.

[0014] (Composition 10) The program of configuration 10 is a program to be executed by an information processing system comprising an application execution unit that executes an application program, an image generation unit that generates an image, and an image output unit that outputs the image generated by the image generation unit to a display screen, wherein the information processing system is to execute the following steps: the image generation unit sequentially executes, within a first unit period, a first process based on processing data at a first timing and a second process including rendering for an application based on the execution result by the application execution unit; if the second process is not completed within the first unit period, interrupt the second process and, within the second unit period following the first unit period, sequentially executes the first process based on the processing data at a second timing later than the first timing and the continuation of the interrupted second process; and the image output unit outputs the image to the display screen without waiting for the unit period to expire, based on the completion of at least the second process. [Brief explanation of the drawing]

[0015] [Figure 1] It is a diagram showing the configuration of the information processing system according to the first embodiment. [Figure 2] It is a diagram showing an example of scheduling. [Figure 3] It is a diagram showing an example of processing when the rendering process for the application is completed before the unit period ends. [Figure 4] It is a diagram showing an example of processing when the rendering process for the application is completed before the unit period ends. [Figure 5] It is a diagram showing an example of processing when the processing for the application is not completed within the unit period. [Figure 6] It is a diagram showing an example of processing when the processing for the application is not completed within the unit period. [Figure 7] It is a flowchart showing the processing flow of the game machine. [Figure 8] It is a diagram showing the configuration of the information processing system according to the second embodiment. [Figure 9] It is a diagram showing the configuration of the information processing system according to the third embodiment. [Figure 10] It is a diagram showing scheduling in the information processing system according to the modification example.

MODE FOR CARRYING OUT THE INVENTION

[0016] Hereinafter, the information processing system of the present embodiment will be described with reference to the drawings. Note that the following description shows only an example of a preferred aspect and is not intended to limit the invention described in the claims.

[0017] (First Embodiment) FIG. 1 is a diagram showing the configuration of the information processing system 1 according to the first embodiment. The information processing system 1 according to the first embodiment is a game machine 10. The game machine 10 includes an input unit 11, an output unit 12, an application execution unit 13, an image generation unit 14, an image output unit 15, an image memory 16, a liquid crystal screen 17, a storage unit 18, and a communication unit 19.

[0018] The input unit 11 has the function of receiving input to the game console 10 through user operation. The input unit 11 is, for example, a touch panel, buttons, an analog stick, etc. The output unit 12 has the function of outputting the results of game processing to the user. The output unit 12 is, for example, a speaker, a haptic device, a display, etc. Note that in Figure 1, a liquid crystal screen 17, which is one of the output units 12, is shown separately.

[0019] The LCD screen 17 supports variable refresh rate (hereinafter referred to as "VRR"). Here, the LCD screen supporting variable refresh rate includes, for example, the LCD screen and the driver IC that controls the LCD screen supporting variable refresh rate. VRR is a technology that synchronizes the refresh rate of the display with the frame rate generated by the graphics card or game console 10. This reduces visual defects such as tearing and stuttering, and provides a smooth and comfortable viewing experience.

[0020] The application execution unit 13 reads the game program (game application program) from the memory unit 18 and executes the read game program. The hardware of the application execution unit 13 is the CPU.

[0021] The image generation unit 14 performs rendering processes for 3D and 2D graphics (rendering processes for applications), as well as physics calculations and animation calculations, based on the execution results of the game program by the application execution unit 13. The image generation unit 14 stores the images generated by the rendering process in the image memory 16. The physics calculations performed by the image generation unit 14 include, for example, calculations of collisions between objects in the game, and calculations that realistically simulate the movement of fluids such as water and smoke.

[0022] The hardware of the image generation unit 14 is a GPU. In addition to application processing based on the execution results of the game program, the image generation unit 14 performs system processing based on the execution results of a system different from the game program. System processing includes, for example, system rendering processing such as drawing the software keyboard, drawing the logo when the application starts, and displaying a pop-up when there is a notification from another device.

[0023] The system processing also includes processing of camera images acquired from the camera and sending them to the communication partner's game console 10. Examples of camera image processing include rendering a virtual background and face detection. After virtual background processing and face detection processing, the image is sent to the communication partner's game console 10, where the image is rendered. The image rendered here is also displayed on the LCD screen 17 of the user's own game console 10. Note that the data sent to the communication partner's game console does not have to be the image itself; it may be data for generating the image. For example, data that identifies the area of ​​the camera image in which a face is visible may be sent. Also, when sending an image, the image of the detected face may be sent separately from the camera image.

[0024] Here, we have explained an example where the generated camera image is displayed on the user's own game console 10, but it is also possible to configure the system so that the image sent to the other party's game console 10 is not displayed on the user's own game console 10.

[0025] The image output unit 15 is a device that transmits image data to the liquid crystal screen 17. An example of the image output unit 15 is a display controller. The image output unit 15 transmits image data stored in the image memory 16 to the liquid crystal screen 17. The image generation unit 14 and the image output unit 15 access the same image memory 16. There is no need to send or receive image data between the image generation unit 14 and the image output unit 15. Here, multiple buffers may be prepared on the memory 16, and parallel processing of reading and writing may be achieved by performing double buffering or triple buffering.

[0026] When the rendering process for the application by the image generation unit 14 is completed, the image output unit 15 transmits a frame signal to the liquid crystal screen 17 and has a function of notifying the liquid crystal screen 17 of the frame generation timing. In response to this, the liquid crystal screen 17 refreshes the screen. Since the liquid crystal screen 17 refreshes the screen in accordance with the timing of the completion of the rendering for the application, tearing can be prevented.

[0027] The scheduling of the processing of the image generation unit 14 in the game machine 10 of the present embodiment will be described. The image generation unit 14 sets a unit period for performing system processing using data necessary for system processing and application processing including rendering processing for the application.

[0028] FIG. 2 is a diagram showing an example of scheduling. The upper part shows the time line of the processing performed by the image generation unit 14, and the lower part shows the timing of refreshing the liquid crystal screen. The system processing and the application processing are each allocated an execution time at a predetermined ratio. For example, the image generation unit 14 executes the system processing and the application processing within a unit period of T seconds. In this example, the system processing takes kT seconds (0 < k < 1) and the application processing takes (1 - k)T seconds.

[0029] In the present embodiment, within the unit period, the processing is executed in the order of system processing and application processing. That is, the system processing is executed at the timing when the unit period starts, and the application processing is executed at the timing when the time allocated to the system processing ends. In the present embodiment, the start of the execution of the application processing is fixed at the timing when the time allocated to the system processing ends. That is, even if the system processing is completed earlier than kT seconds, the application processing starts after kT seconds from the start of the unit period. As a result, the time available for the application processing within the unit period (in the example of FIG. 2, (1 - k)T seconds) does not change.

[0030] The length of the unit period can be selected by the game program based on its estimation of the rendering load. For example, the unit period can be 1 / 60th of a second, 1 / 30th of a second, or 1 / 120th of a second, and the game program will select the appropriate unit period from among these. For example, it will estimate the rendering load and lengthen the unit period if the rendering process is heavy, and shorten it if the rendering process is light. The length of the unit period may also be set manually by the user.

[0031] In the game console 10 of this embodiment, the image generation unit 14 performs system processing for a predetermined time within a unit period (kT seconds in the example of Figure 2). When the unit period of T seconds ends, the next unit period starts even if the application processing is not completed, and system processing is executed. Conversely, if the application processing is completed within the unit period, the image generation unit 14 does not wait for the end of the unit period, but sends a frame signal to the liquid crystal screen 17 to start the next unit period.

[0032] Figures 3 and 4 show an example of the process when the application rendering process is completed before the end of the unit period. As shown in Figure 3, if the application rendering process is completed before the end of the unit period, the image generation unit 14 sends a frame signal to the liquid crystal screen 17 without waiting for the unit period to expire. As a result, the liquid crystal screen 17 is refreshed, and the image data rendered by the application rendering process is displayed on the liquid crystal screen 17. If system processing, including system rendering processing, is executed, an image based on the results of the application rendering process and the system rendering process is displayed. Figure 4 shows the process following Figure 3. As shown in Figure 4, the image generation unit 14 starts a new unit period after sending the frame signal.

[0033] Figure 5 shows an example of the process when the application processing does not finish within a unit period. As shown in Figure 5, the system process Sy1 and the application process Ap1 are executed in the first unit period. If the application process Ap1 does not finish, the image generation unit 14 starts the next unit period without sending a frame signal to the liquid crystal screen 17. In other words, if the application process Ap1 does not finish, the application process Ap1 is interrupted and a new unit period is started, and the application process Ap1 is continued in the next unit period. In the next unit period, the system process Sy2 is executed, followed by the continuation of the application process Ap1. Here, the system process Sy2 may be a different process from or the same as the system process Sy1. When the application process Ap1 finishes, the image generation unit 14 sends a frame signal to the liquid crystal screen 17. As a result, the liquid crystal screen 17 is refreshed, and the image data rendered by the application rendering process is displayed on the liquid crystal screen 17. After sending the frame signal, the image generation unit 14 starts a new unit period.

[0034] In the example shown in Figure 5, the application processing Ap1 terminates simultaneously with the end of the second unit period. However, as shown in Figure 6, if the resumed application processing Ap1 terminates in the middle of the next unit period, the image generation unit 14 sends a frame signal to the liquid crystal screen 17 without waiting for the unit period to expire.

[0035] In both Figure 5 and Figure 6, if system rendering is performed as part of the system processing, the system rendering performed in the later unit period becomes valid, and the system rendering performed in the previous unit period becomes invalid. In other words, the image generated by the system rendering in the previous unit period is not reflected on the LCD screen 17. In Figures 5 and 6, the application processing Ap1 is completed in two unit periods, but there are cases where application processing takes three or more unit periods. In such cases, the system rendering performed in the last unit period becomes valid.

[0036] Furthermore, with respect to system processing related to camera images, system processing other than the last executed system processing may also be valid. For example, camera image processing may be performed over multiple unit periods, such as interrupting camera image processing in the first unit period and resuming it from where it left off in the second unit period.

[0037] Figure 7 is a flowchart showing the processing flow of the game console 10. The image generation unit 14 of the game console 10 determines whether or not there is any executable system processing (S10). If there is executable system processing (YES in S10), the image generation unit 14 executes the system processing from the start of the unit period (S11). If there is no system processing to be executed (NO in S10), the process proceeds to determine whether the allocated time for system processing has elapsed (S12). Note that if there is no system processing to be executed at this time, no system processing is performed. Here, the absence of system processing refers to cases such as when there are no notifications from other terminals and there is no processing related to camera images to be sent to other terminals.

[0038] The image generation unit 14 executes system processing until the time allocated for system processing within the unit period has elapsed (until it is determined to be NO in S12). The time allocated for system processing is, for example, kT seconds from the start of the unit period in Figure 2.

[0039] When the time allocated for system processing has elapsed (YES in S12), the image generation unit 14 executes rendering processing for the application (S13). When the rendering processing for the application is completed (YES in S14), the image generation unit 14 sends a frame signal to the liquid crystal screen 17 (S15), and then the image output unit 15 transfers image data to the liquid crystal screen 17 (S16). The liquid crystal screen 17 refreshes the displayed image using the image data received from the image output unit 15 (S17). The process described here corresponds to the completion timing process in Figure 3 above. After that, the game machine 10 resets the unit period (S20) and terminates the process.

[0040] If the rendering process for the application is not completed (NO in S14), the image generation unit 14 determines whether the unit period has ended (S18). If the unit period has not ended (NO in S18), it continues to execute the rendering process for the application (S13) and determines whether the rendering process for the application has been completed (S14).

[0041] If the unit period ends before the application rendering process is completed (YES in S18), the application rendering process is interrupted (S19), the unit period is reset (S20), and the processing for that unit period ends. The processing for one unit period is completed by executing the actions shown in Figure 7 to the end. As described above, the processing ends either when the unit period ends (flow from S18) or when the application rendering process is completed (flow from S14), but in either case the unit period is reset and the game console 10 executes the flow shown in Figure 7 again from the beginning.

[0042] The game console 10, which is the information processing system 1 of the first embodiment, has been described above. In the game console 10 of this embodiment, the image generation unit 14 performs system processing and application processing within a unit period, so that system processing is always executed at least every unit period. Although rendering processing for applications may take time, system processing can be executed at an appropriate timing even in such cases. If rendering for applications is completed early, the LCD screen 17 is refreshed without waiting for the end of the unit period, enabling smooth screen display.

[0043] In this embodiment, a game device and its processing method have been described, but the scope of the present invention also includes a program that schedules the image generation unit 14 and the image output unit 15 to execute the above-described processing method.

[0044] (Second Embodiment) Figure 8 shows the configuration of the information processing system 2 of the second embodiment. The information processing system 2 of the second embodiment includes a game console 10 and a monitor 20. In the second embodiment, the game console 10 is used by connecting it to an external monitor 20 such as a TV. The external monitor 20 supports VRR.

[0045] The basic configuration of the game console 10 is the same as that of the game console 10 in the first embodiment. In the game console 10 of the second embodiment, the image output unit 15 transmits image data to an external monitor 20 instead of the LCD screen 17. In addition, when the rendering process for the application is completed, the image generation unit 14 transmits a frame signal to the monitor 20 indicating that the rendering process is complete.

[0046] Even when the image display device is an external monitor 20 that supports VRR, the game console 10 can execute system processing at the appropriate timing, just as in the first embodiment.

[0047] If the monitor 20 does not support VRR, the frame will not be refreshed even if a frame signal is sent, so the same processing as the game console 10 in the first embodiment (processing shown in Figures 2 to 7) cannot be performed. The game console 10 obtains EDID (Extended Display Identification Data) from the connected monitor 20, determines whether the monitor 20 supports VRR, and performs processing using a unit period if it is determined that the monitor 20 supports VRR.

[0048] (Third embodiment) Figure 9 shows the configuration of the information processing system 3 of the third embodiment. In the information processing system 3 of the third embodiment, the game machine 10 is configured to be able to communicate with other game machines 30. The basic configuration of the game machine 10 is the same as that of the game machine 10 described in the first embodiment. The other game machine 30 may be the same as the game machine 10 described in the first embodiment, or it may have a different configuration. The connection between the game machine 10 and the other game machine 30 may be via P2P communication or via a server.

[0049] In the information processing system 3 of this embodiment, the game machine 10 and another game machine 30 can send and receive data related to the progress of the game and camera image data taken by the user. The camera image data is processed by the system.

[0050] Let's explain camera processing here. For example, if the unit period is 60 [fps] (T = 1 / 60 second), one camera process can be divided into four system processes. In this case, the camera processing will be performed at 15 [fps]. Note that if the application processing finishes early, the system processing will be performed ahead of schedule, so the camera processing for one frame will finish earlier than 15 [fps].

[0051] When the other game console 30 receives camera image data transmitted from game console 10, it displays the camera image on the LCD screen provided in the game console 30. If game console 30 is connected to an external display device such as a monitor, the other game console 30 may also display the received camera image on the external display device.

[0052] Even when connected to other game consoles 30 and processing camera image data intended for those other game consoles 30, the camera processing will never be performed at a speed slower than 15 fps. Note that while Figure 9 shows only one other game console 30, there may be multiple other game consoles 30.

[0053] Although the information processing system of the present invention has been described in detail with reference to embodiments above, the information processing system of the present invention is not limited to the embodiments described above.

[0054] In the above-described embodiment, time is allocated within a unit period for executing system processing, and an example was described in which application processing is executed when the time allocated for system processing has elapsed. However, if the system processing is completed earlier, the time allocated for system processing may be used for application processing.

[0055] Figure 10 shows the scheduling in the information processing system according to a modified example. As shown in Figure 10, the unit period has time allocated to system processing (kT seconds) and time allocated to application processing ((1-k)T seconds). Therefore, the image generation unit 14 cannot perform system processing beyond the time allocated for system processing, as in the embodiment described above, and terminates system processing when the allocated system processing time ends.

[0056] On the other hand, as shown in the scheduling in Figure 10, if system processing is completed before the end of the system processing allocation time, the modified information processing system will execute application processing from the moment the system processing is completed. If there is no system processing to be executed at the start of the unit period, application processing will start immediately. This configuration eliminates the need to wait for the system processing allocation time to elapse after the system processing is completed, thus speeding up the completion time of application processing and, consequently, improving the frame rate.

[0057] The example shown in Figure 10 describes a case where system processing is completed before the end of the allocated system processing time. However, the allocation of system processing time and application processing time may be changed depending on the application's execution status. For example, if the amount of processing required for the application is small, such as when the application is not running or is running in the background, the allocated system processing time per unit period may be increased.

[0058] Furthermore, although the above-described embodiment explained the case where the processing order within a unit period is system processing followed by application processing, the processing order may also be application processing first and system processing second.

[0059] Furthermore, although the above-described embodiment shows an example where the time allocated to application processing is longer than the time allocated to system processing, the time allocated to system processing may also be longer than the time allocated to application processing.

[0060] Furthermore, although the above-described embodiment shows an example in which two processes, system processing and application processing, are executed during a unit period, three or more processes may be executed during a unit period. [Explanation of Symbols]

[0061] 1-3 Information Processing Systems 10 game consoles 11 Input section 12 Output section 13. Application execution unit 14 Image generation unit 15 Image output section 16 Image memory 17 LCD screen 18 Memory section 19 Communications Department 20 monitors 30 Other game consoles

Claims

1. The application execution unit that runs the application program, An image generation unit that generates images, An image output unit that outputs the image generated by the image generation unit to a display screen, Equipped with, The image generation unit, Within the first unit period, a first process based on processing data at a first timing and a second process including rendering for the application based on the execution result by the application execution unit are executed sequentially. If the second process is not completed within the first unit period, the second process is interrupted, and within the second unit period following the first unit period, the first process based on the processing data at a second timing later than the first timing, and the continuation of the interrupted second process are executed sequentially. The image output unit outputs the image to the display screen without waiting for the unit period to expire, based on the completion of at least the second process. Information processing system.

2. The processing data is updated according to the execution result of a system process different from the application program. The first process includes a system rendering process corresponding to the result of the system processing, The information processing system according to claim 1, wherein the image output unit outputs the image to the display screen without waiting for the expiration of the unit period, based on the completion of the system rendering process and the second process.

3. The aforementioned information processing system is capable of communicating with other information processing devices and includes an information processing device equipped with the aforementioned display screen, The aforementioned information processing system is The image generation unit performs the first processing, including the processing for transmission, The system further includes a communication unit that transmits information based on the results of the transmission process to the other information processing device, The information processing system according to claim 2, wherein the image based on the results of the transmission process is not output to the display screen of the information processing system, but is output to the display screen of the other information processing device.

4. The information processing system according to claim 1, wherein the image generation unit performs the first process before the second process in the unit period.

5. The information processing system according to claim 4, wherein the second process is not performed until a predetermined time has elapsed since the first process was performed.

6. The first process is completed when either the processing completion condition is met, or a predetermined time has elapsed since the first process was executed. The information processing system according to claim 4, wherein the second process is executed without waiting for the predetermined time to elapse when the first process is completed based on the fulfillment of the processing completion conditions.

7. The information processing system according to claim 1, wherein the length of the unit period is determined according to the application program executed by the application execution unit.

8. The aforementioned information processing system includes an information processing device and a display device, The information processing device comprises the application execution unit, the image generation unit, and the image output unit. The information processing system according to claim 1, wherein the display device comprises the display screen.

9. An information processing method in an information processing system comprising an application execution unit that executes an application program, an image generation unit that generates an image, and an image output unit that outputs the image generated by the image generation unit to a display screen, The image generation unit sequentially performs, within a first unit period, a first processing based on processing data at a first timing, and a second processing including rendering for the application based on the execution result by the application execution unit. If the second process is not completed within the first unit period, the second process is interrupted, and within the second unit period following the first unit period, the first process based on the processing data at a second timing later than the first timing, and the continuation of the interrupted second process are executed sequentially. The image output unit, based on the completion of at least the second process, outputs the image to the display screen without waiting for the unit period to expire, An information processing method comprising the following:

10. A program to be executed by an information processing system comprising an application execution unit that executes an application program, an image generation unit that generates an image, and an image output unit that outputs the image generated by the image generation unit to a display screen, wherein the information processing system The image generation unit sequentially performs, within a first unit period, a first processing based on processing data at a first timing, and a second processing including rendering for the application based on the execution result by the application execution unit. If the second process is not completed within the first unit period, the second process is interrupted, and within the second unit period following the first unit period, the first process based on the processing data at a second timing later than the first timing, and the continuation of the interrupted second process are executed sequentially. The image output unit, based on the completion of at least the second process, outputs the image to the display screen without waiting for the unit period to expire, A program that executes the command.

Images