Information Processing Apparatus, Information Processing Method, and Program

Abstract

To provide an information processing device, an information processing method, and a program capable of reducing delay from operation to drawing.SOLUTION: An information processing device including a processor draws an image frame by frame, buffers the image, and outputs a buffered frame at a predetermined timing. The processor executes waiting processing for waiting for the frame drawing processing until instructed timing, and controls a start of the frame drawing processing after the waiting processing.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, since the processing speeds of processors such as CPUs and GPUs (Graphics Processing Units) have been improving, it has become possible to draw animations in real time.

[0003] For example, in an application such as a game, it is also possible to repeatedly change the viewpoint of a user character in a virtual game space according to an operation input by the user, and draw and output an image of the game space seen from the changed viewpoint.

[0004] Also, in such a case, it has become common to draw (buffer) a frame corresponding to the screen to be displayed and output before output, and perform display output at a predetermined timing after the drawing is completed.

Summary of the Invention

Problems to be Solved by the Invention

[0005] Under such circumstances, as illustrated in FIG. 8, user input until the time point A when drawing of a certain frame to be displayed in the future starts is received by the CPU, and drawing processing of the frame to be displayed in the future is performed based on the user input up to this point. In this drawing process of the CPU, an instruction to draw is given to the GPU, and the actual drawing of the frame image is performed by the GPU. The GPU holds this frame (buffers) until a predetermined future timing B (flip timing), and output processing (image output) to a display device or the like is performed at the predetermined timing B.

[0006] Therefore, for the user, the display corresponding to the operation performed at timing A will be delayed by time T until timing B. Such a delay is not preferable in an application that requires real-time processing such as a game application.

[0007] The present invention has been made in view of the above circumstances, and one of its objects is to provide an information processing apparatus, an information processing method, and a program that can shorten the delay from such an operation to drawing.

Means for Solving the Problem

[0008] One aspect of the present invention that solves the problems of the above conventional example is an information processing apparatus that includes a processor, draws and buffers an image for each frame, and outputs the buffered frames at predetermined timings, wherein the processor executes a standby process of waiting for the drawing process of the frame until the instructed timing, and starts controlling the execution of the drawing process of the frame after the standby process.

Effect of the Invention

[0009] According to the present invention, by accepting user input while waiting for the drawing process, the delay from the operation to the drawing can be shortened.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Embodiments for Carrying Out the Invention

[0011] Embodiments of the present invention will be described with reference to the drawings. As illustrated in FIG. 1, the information processing apparatus 1 according to the embodiment of the present invention includes a control unit 11 as a processor, a storage unit 12, an operation control unit 13, and a display control unit 14, and more preferably includes a communication unit 15.

[0012] Further, a peripheral device 2 equipped with a GPU 21 which is a graphic processor may be connected to the display control unit 14 of the information processing apparatus 1. Note that the information processing apparatus 1 and the peripheral device 2 may be integrally configured as an information processing system.

[0013] Furthermore, a game controller C, a keyboard, a mouse, and other operation devices 3 are connected to the information processing apparatus 1 by wire or wirelessly. A display device D is connected to the peripheral device 2.

[0014] The control unit 11 is a program control device such as a CPU and realizes the processor of the present invention. This control unit 11 operates according to a program stored in the storage unit 12. In an example of the present embodiment, when performing the drawing process of a frame, this control unit 11 executes a standby process of waiting for the drawing process of the frame until the instructed timing, and starts to control the execution of the drawing process of the frame after the standby process. The operation of this control unit 11 will be described in detail later.

[0015] The storage unit 12 is a memory device, a disk device, etc., and holds programs executed by the control unit 11. The programs held in this storage unit 12 may be stored and provided in a computer-readable and non-transitory recording medium. The storage unit 12 of the present embodiment also operates as a working memory of the control unit 11.

[0016] The operation control unit 13 receives information representing the content of the user's operation from an operation device 3 such as a game controller. The operation control unit 13 outputs the information representing the received operation content to the control unit 11.

[0017] The display control unit 14 receives information such as a drawing instruction generated by frame drawing processing from the control unit 11, and outputs the information such as the drawing instruction to the peripheral device 2.

[0018] The GPU 21 of the peripheral device 2 receives an instruction for drawing a frame input from the control unit 11 and holds it in a command buffer. When drawing becomes possible, according to the held drawing instruction, the GPU 21 draws the frame image in the image buffer. This GPU 21 also outputs the frame drawn in the image buffer to the display device at a predetermined timing (for example, the VSYNC timing of the display device). In an example of the present embodiment, the GPU 21 has a pair of command buffers and a pair of image buffers (so-called double buffer). Further, unique identification information (frame number) is associated with each frame to be drawn, and may be used in the control unit 11 and the GPU 21.

[0019] An example of the operation device 3 is a game controller C, which receives a user's instruction operation and outputs information representing the content of the user's operation to the information processing device 1 by wire or wirelessly.

[0020] Next, an operation example of the control unit 11 of the present embodiment will be described. By executing the program held in the storage unit 12, the control unit 11 of the present embodiment realizes a functional configuration including an input reception unit 41, a standby processing unit 42, a processing start control unit 43, and an application processing unit 44, as illustrated in FIG. 2. Here, a drawing processing unit 441 is functionally included in the application processing unit 44.

[0021] The input reception unit 41 receives operation information representing the content of the user's operation on the operation device 3 via the operation control unit 13. This input reception unit 41 buffers and holds the received operation information in the order of reception, and outputs it to the application processing unit 44 in that order. Also, this input reception unit 41 receives operation information representing the content of the operation performed by the user even while the standby processing unit 42, which will be described later, is waiting for the drawing processing by the application processing unit 44 (during standby processing) (receiving input from the user).

[0022] The standby processing unit 42 waits for the frame drawing processing by the application processing unit 44 until the instructed timing (hereinafter, this timing is called the drawing start timing for distinction). In an example of the present embodiment, this standby processing unit 42 may determine the drawing start timing by a predetermined method based on the future timing (hereinafter, this timing is called the reference timing for distinction) at which the frame to be displayed in the future (conveniently called the target frame) is displayed and output (hereinafter called flip). Here, the target frame is a frame to be drawn in the future based on the drawing instruction generated by the drawing processing unit 441 of the application processing unit 44 by the drawing processing for which the standby processing unit 42 is waiting.

[0023] Specifically, the standby processing unit 42 receives, as information instructing a condition for ending the standby processing from the application that executes this drawing processing, the number of frames n (n is an integer of 1 or more) from the target frame Ft to another frame Fr output in advance, and the elapsed time (hereinafter called additional time) t from the timing at which the other frame Fr is flipped.

[0024] The standby processing unit 42 receives information indicating the condition and waits until the indicated timing arrives. That is, as illustrated in FIG. 3, at point A when the application attempts to start drawing the target frame Ft to be displayed in the future, the standby processing unit 42, as information indicating the condition for ending the standby processing, receives an input of the number of frames n from the target frame Ft to another frame Fr output prior thereto (hereinafter referred to as the specified number of frames; n is an integer of 1 or more) and the elapsed time t from the timing when the other frame Fr is flipped (hereinafter referred to as the additional time). In the example here, n = 2 and the additional time t = 1 millisecond.

[0025] The standby processing unit 42 of the control unit 11 (CPU in FIG. 3) monitors the flip state of the frame by the GPU 21 and waits until the frame Fr, which is two frames before the target frame Ft in terms of the specified number of frames n = 2, among the other frames flipped before the flip timing (future point B) of the target frame Ft, is flipped.

[0026] When the frame Fr is flipped, the standby processing unit 42 waits for an additional time (1 millisecond in this example) from that timing C. Then, when the time point C′ at which the additional time has elapsed arrives, the standby processing unit 42 notifies the processing start control unit 43 of the end of the standby processing.

[0027] The processing start control unit 43 instructs the drawing processing unit 441 of the application processing unit 44 to start drawing the target frame Ft that has been waiting. Thereby, the drawing processing is started and controlled from the time point C′.

[0028] The application processing unit 44 executes the processing of the application program. In the example here, based on the operation information representing the content of the operation performed by the user, it executes the drawing processing of the frame and the like.

[0029] The application program executed by this application processing unit 44 may be any program such as a game program or a simulator.

[0030] When the drawing processing unit 441 of this application processing unit 44 finishes the drawing processing of the previous frame, it waits until it receives an instruction from the processing start control unit 43 to start drawing the next frame. Then, when it receives an instruction to start drawing this frame, it acquires the operation information input by the user that the input reception unit 41 has received up to that point (point C' in FIG. 3 in this example), and executes the drawing process using the operation information.

[0031] The drawing process by the control unit 11 here is actually a process of generating a drawing instruction for the GPU 21. After this drawing process, the actual frame is drawn by the GPU 21.

[0032] The GPU 21 of the peripheral device 2 holds the drawing instruction input from the control unit 11 which is a CPU in the command buffer. If there is an image buffer that is not before display or during display (if there is an empty buffer), it draws (buffers) the frame image into the empty buffer according to the drawing instruction held in the command buffer. Then, it outputs (flips) the buffered frame image to a display device or the like at a predetermined timing (for example, the timing of the next VSYNC).

[0033] Note that the timing of this flip may be instructed from the control unit 11 of the information processing apparatus 1, or may be controlled on the GPU 21 side.

[0034] [Operation] This embodiment basically has the above configuration and operates as follows. In an example of this embodiment, the information processing apparatus 1 of the information processing system executes a game application and performs drawing of the game space according to the user's operation.

[0035] This information processing system also includes a peripheral device 2 including a GPU 21, and this GPU performs rendering and display on a display device using a pair of command buffers and a pair of rendering buffers.

[0036] During the execution of a game application, the information processing device 1 repeatedly executes frame rendering and display as follows. That is, as illustrated in FIG. 3, the information processing device 1 repeatedly accepts operation information representing the content of a user's operation on the operation device 3 via the operation control unit 13.

[0037] Also, as part of the game application processing, at the timing (time point A) when the information processing device 1 attempts to start rendering the target frame Ft to be displayed in the future, the information processing device 1 instructs the waiting process to end under the condition of the number of specified frames n (n is an integer of 1 or more) from the target frame Ft to another frame Fr output in advance, and the additional time t from the timing when the other frame Fr is flipped, for the waiting process that waits for the rendering timing.

[0038] In the example of FIG. 3, this instruction sets the number of specified frames n = 2 and the additional time t = 1 millisecond (the display period of one frame, for example, a time shorter than the VSYNC period).

[0039] As illustrated in FIG. 3, the information processing device 1 executes a waiting process as various services (API: Application Program Interface) performed on the application program, monitors the state of frame flipping by the GPU 21, and waits until a frame Fr that is two frames before the target frame Ft among other frames flipped before the flipping timing (future time point B) of the target frame Ft is flipped.

[0040] In this standby process, when the frame Fr is flipped, the information processing apparatus 1 waits for an additional time (1 millisecond in this example) from the timing C. Then, when the time point C' at which the additional time has elapsed arrives, the information processing apparatus 1 instructs to start drawing the target frame Ft that has been waiting for the processing of the application program.

[0041] The information processing apparatus 1 returns to the processing of the application program, and based on the operation information representing the content of the operation received via the operation control unit 13 until this timing C', executes frame drawing processing and the like.

[0042] As shown in the example of FIG. 3, although the image in the buffer of the drawing destination is still in the state of not being displayed at this timing C', it is possible to generate a drawing instruction in the CPU and buffer the drawing instruction to the command buffer in the GPU 21.

[0043] Therefore, the information processing apparatus 1 starts the drawing process (generation of drawing instructions) in the CPU from this timing C'. This process is executed as the processing of the application program. In the example here of the present embodiment, an instruction to draw an image in the game space is generated using the operation information input from the user until the timing C'. For example, the movement of the user character arranged in the game space is controlled by the search information input from the user until this timing C'.

[0044] The information processing apparatus 1 outputs the generated drawing instruction to the GPU 21. The GPU 21 holds the drawing instruction input from the information processing apparatus 1 in the command buffer, and at a timing after the timing when an image buffer that is not before or during display occurs (the timing when the frame immediately before the target frame is flipped), draws (buffers) the frame image to the image buffer that has become an empty buffer according to the drawing instruction held in the command buffer. Then, the image of the buffered frame is output to the display device or the like at the next VSYNC timing (B).

[0045] According to this example, compared with the time T between time point A and time point B in the conventional example shown in FIG. 8, the corresponding time T' is shortened, and the image flipped at time point B reflects the content that the user has operated relatively recently. Thus, according to this embodiment, the delay from operation to drawing can be shortened.

[0046] [When it takes a relatively long time for frame drawing] Note that in the example of FIG. 3, the time required for the GPU 21 to draw a frame is well below the display period of the frame image. However, when the time required for the GPU 21 to draw a frame is longer than this, increase the specified number of frames n or shorten the additional time t. Then, among the frame drawings by the GPU 21, the capture of the drawing instruction from the GPU 21 to the command buffer is performed before the drawing buffer to be drawn becomes an empty buffer (FIG. 4).

[0047] In this example, since the period from the completion of the capture of the drawing instruction to the drawing buffer becoming an empty buffer becomes a stall (a state where nothing is done), this stall period is made as short as possible (0 if possible), and the drawing to the drawing buffer is completed before the timing of flipping the target frame drawn in the drawing buffer. The specified number of frames n and the additional time t are determined accordingly. Such settings of the specified frame n and the additional time t can be made experimentally, for example, and the developer of the application program may design to adaptively set them according to, for example, the number of polygons to be displayed in the processing of the application program.

[0048] [Timeout] Further, when the timing at which a frame that is flipped prior to the target frame output at a reference timing during the standby process arrives, and this frame is flipped L (L is an integer such that 1 ≦ L < n and is predetermined) earlier than the target frame, the control unit 11 of the information processing apparatus 1 may interrupt the standby process and start executing the drawing process.

[0049] In yet another example, in the standby process, when the predetermined timeout time has elapsed since the start of the standby process, the control unit 11 of the information processing apparatus 1 may interrupt the standby process at that time and start executing the drawing process. Such a case may occur when the flip timing becomes irregular, etc.

[0050] [End condition of standby process] In the description so far, the end condition of the standby process has been set according to the relative number of frames (designated number of frames) from the reference target frame, but the present embodiment is not limited to this.

[0051] For example, the end condition of this standby process may be determined by the identification information of the frame. As a specific example, the end condition may be determined by the frame number (N) which is the identification information of the frame and the additional time t from the timing when the frame FN of the frame number N is flipped. In this example, the control unit 11 that executes the processing of the application program instructs these frame numbers (N) and the additional time t to the standby process that waits for the drawing timing.

[0052] The control unit 11 that executes the standby process accepts the frame number (N) and the additional time t, starts the standby process, monitors the flip state of the frame by the GPU 21, and waits until the flip timing of the frame of the frame number N.

[0053] In this standby process, when the frame of the frame number N is flipped, the control unit 11 of the information processing apparatus 1 waits for an additional time t from that timing. Then, when the time point when the additional time t has elapsed arrives, the control unit 11 instructs the application program processing to start drawing the target frame that has been waiting.

[0054] The information processing apparatus 1 returns to the processing of the application program, and executes frame drawing processing and the like based on the operation information representing the content of the operation received via the operation control unit 13 until the timing when the start of this drawing is instructed.

[0055] [Query to GPU] The control unit 11 of the information processing apparatus 1 may inquire about the processing status of the GPU 21 of the peripheral device 2 at each predetermined inquiry timing. Specifically, when the control unit 11 does not control the timing of image flipping, etc., the control unit 11 inquires the GPU 21 about the identification information of the frame that was flipped most recently.

[0056] In this example, the GPU 21 outputs the identification information of the frame that was flipped most recently to the control unit 11 in response to the inquiry from the control unit 11. The control unit 11 refers to the identification information received as a response to this inquiry, and when the frame identified by the identification information corresponds to the frame specified by the relative specified number of frames from the target frame, or when the frame identified by the identification information corresponds to the frame specified by the frame number (N), the control unit 11 waits for an additional time t from the time of flipping of the frame and then ends the standby process, and performs start control of the execution of the drawing process.

[0057] [Parallel operation of CPU and GPU] Also, the drawing process (generation process of drawing instructions) by the control unit 11 and the drawing process (accumulation of drawing instructions and actual drawing process of frames) by the GPU 21 may be performed in parallel. In this case, the drawing process by the GPU 21 is started before the end of the drawing process of the control unit 11 (Fig. 5). Fig. 5 shows an example where the relative specified number of frames n = 1 from the target frame. By operating the control unit 11 and the GPU 21 in parallel during the display period of one frame in this way, the period until the corresponding image is displayed from the user's operation can be shortened without degrading the performance.

[0058] This operation is the same when the end condition of the waiting process is specified by the frame number (N) (Figure 6). Figure 6 shows an example where the frame number N = 2.

[0059] [Example of triple buffer] Furthermore, the number of command buffers and drawing buffers is not limited to 2 (double buffer), and there may be 3 (triple buffer) or more. In this case, if the above-described waiting process of the present embodiment is not performed, the delay in the timing of displaying in response to the user's operation becomes larger than that in the example of the double buffer.

[0060] On the other hand, for example, if the waiting process is performed until a time point t after the flip of the frame n = 3 before the target frame, as illustrated in FIG. 7(a), the delay T' in the timing of displaying in response to the user's operation is relatively shortened. However, in this case, the effect of the triple buffer is reduced.

[0061] In such an example, in the processing of the application program, for example, according to the scene of the game, the specified number of frames n or the frame number N, and the additional time t are adaptively adjusted. Thereby, as in the example of FIG. 7(a), for example, by making the specified number of frames n relatively small and reducing the effect as a triple buffer, the timing delay T' is shortened, or as illustrated in FIG. 7(b), for example, by making the specified number of frames n relatively large (n = 4 in FIG. 7(b)), it is possible to control whether to increase the timing delay T' or secure the number of drawing buffers (it becomes possible to increase the time taken for the drawing process).

[0062] The former is effective in a scene where the display delay from the operation is a problem, such as a fighting scene, and the latter is effective in a scene where the detail of the display content is more of a problem than the display delay from the operation, such as a landscape description scene.

[0063] [Processing of accepting operations] In the description so far, the process of receiving operation information by the control unit 11 may be processed (parallel processed) in a thread different from the drawing process, or the operation information may be processed in the same thread as the drawing process. Whether to do so may be selected by the application program or determined based on the conditions of the hardware to be executed.

Description of Signs

[0064] 1 Information processing apparatus, 2 Peripheral apparatus, 3 Operating apparatus, 11 Control unit, 12 Storage unit, 13 Operation control unit, 14 Display control unit, 15 Communication unit, 21 GPU, 41 Input reception unit, 42 Standby processing unit, 43 Processing start control unit, 44 Application processing unit, 441 Drawing processing unit.

Claims

An information processing apparatus connected to a peripheral device equipped with a graphic processor, comprising a processor, which draws and buffers an image frame by frame and outputs the buffered frame at a predetermined timing, wherein: the processor executes a waiting process for waiting for the frame drawing process until the instructed timing, after the waiting process, starts controlling the execution of the frame drawing process, the frame drawing process by the processor includes a process of generating a drawing instruction for the graphic processor, the graphic processor of the peripheral device draws and buffers the image of the frame according to the drawing instruction, and outputs the buffered frame at a predetermined timing, the timing at which the processor waits is determined using the timing at which a frame output before the target frame by a specified number of frames is output, based on the timing at which the frame drawn according to the generated drawing instruction is output as the target frame. An information processing apparatus connected to a peripheral device equipped with a graphic processor, comprising a processor, which draws and buffers an image frame by frame and outputs the buffered frame at a predetermined timing, wherein: the processor executes a waiting process for waiting for the frame drawing process until the instructed timing, after the waiting process, starts controlling the execution of the frame drawing process, the frame drawing process by the processor includes a process of generating a drawing instruction for the graphic processor, the graphic processor of the peripheral device draws and buffers the image of the frame according to the drawing instruction, and outputs the buffered frame at a predetermined timing, the timing at which the processor waits is determined based on the timing at which the frame drawn according to the generated drawing instruction is output. The instruction of the timing is performed based on the number of frames n (n is an integer of 1 or more) from the frame output at the reference timing to another frame output prior to the frame, and the elapsed time t from the timing at which the other frame is output. In the standby process, the processor waits until the time t has elapsed from the output timing of another frame that is n frames before the frame, for the drawing process of the frame output at the reference timing, as the instructed timing.

3. The information processing apparatus according to claim 2, wherein the number of frames n is an integer of 2 or more, and when the output timing of another frame that is a predetermined L (1 ≦ L < n) frames before the frame output at the reference timing arrives during the standby process of the drawing process, the processor interrupts the standby process even before the time t has elapsed from the output timing of another frame that is n frames before the frame, and starts controlling the execution of the drawing process.

4. The information processing apparatus according to any one of claims 1 to 3, wherein the processor inquires about the processing status of the graphic processor of the peripheral device at each predetermined inquiry timing.

5. An information processing apparatus including a processor, which draws and buffers an image frame by frame, and outputs the buffered frames at predetermined timings, wherein the processor executes a standby process of waiting for the drawing process of the frame until the instructed timing, starts controlling the execution of the drawing process of the frame after the standby process, and when the predetermined timeout time has elapsed from the start of the standby process of waiting for the drawing process of the frame, the processor interrupts the standby process at that time and starts controlling the execution of the drawing process.

6. The information processing apparatus according to any one of claims 1, 2, 3, and 5, wherein the processor accepts an input from a user during the standby process of the drawing process, and in the drawing process, causes the drawing process to be performed using the input accepted during the standby process.

7. An information processing apparatus including a processor and a graphic processor, wherein the processor executes a standby process of waiting for the drawing process of the frame until the instructed timing, After the standby process, start controlling the execution of the frame drawing process. The frame drawing process by the processor includes a process of generating a drawing instruction for the graphic processor. The graphic processor: Draws and buffers an image of a frame according to the drawing instruction, and outputs the buffered frame at a predetermined timing. An information processing apparatus in which the timing at which the processor waits is determined using the timing at which a frame output a specified number of frames before the target frame is output, based on the timing at which the target frame is output with the frame drawn according to the generated drawing instruction as the target frame.

8. Using an information processing apparatus connected to a peripheral device equipped with a graphic processor, equipped with a processor, drawing and buffering an image frame by frame, and outputting the buffered frame at predetermined timings, by the processor executes a standby process of waiting for the frame drawing process until the indicated timing, after the standby process, starts controlling the execution of the frame drawing process, the frame drawing process by the processor includes a process of generating a drawing instruction for the graphic processor, the graphic processor of the peripheral device draws and buffers an image of a frame according to the drawing instruction, and outputs the buffered frame at a predetermined timing, An information processing method in which the timing at which the processor waits is determined using the timing at which a frame output a specified number of frames before the target frame is output, based on the timing at which the target frame is output with the frame drawn according to the generated drawing instruction as the target frame.

9. Using an information processing apparatus equipped with a processor, drawing and buffering an image frame by frame, and outputting the buffered frame at predetermined timings, by the processor, executes a standby process of waiting for the frame drawing process until the indicated timing, after the standby process, starts controlling the execution of the frame drawing process, An information processing method in which when a predetermined timeout time has elapsed since the start of the standby process of waiting for the frame drawing process, the standby process is interrupted at that point and the execution of the drawing process is started and controlled. An information processing apparatus connected to a peripheral device having a graphic processor, which draws and buffers an image frame by frame and outputs the buffered frame at a predetermined timing, means for executing a waiting process for waiting for the frame drawing process until the instructed timing; means for starting and controlling the execution of the frame drawing process after the waiting process; functioning as; the frame drawing process includes a process of generating a drawing instruction for the graphic processor; the graphic processor of the peripheral device draws and buffers the frame image according to the drawing instruction, and outputs the buffered frame at a predetermined timing; a program for determining the waiting timing by using the timing at which a frame output a specified number of frames before the target frame is output, with the timing at which the frame drawn according to the generated drawing instruction is output as the target frame as a reference. An information processing apparatus that draws and buffers an image frame by frame and outputs the buffered frame at a predetermined timing, means for executing a waiting process for waiting for the frame drawing process until the instructed timing; starting and controlling means for executing the frame drawing process after the waiting process; functioning as; a program for interrupting the waiting process at that time and starting and controlling the execution of the drawing process when a predetermined timeout time has elapsed since the start of the waiting process for waiting for the frame drawing process.

Images