Data processing systems, data processing methods, and computer programs

Abstract

A data processing system 10 is provided, which includes a GPU 14 and a switching command unit 44. The GPU 14 executes multiple data generation processes corresponding to multiple applications executed in parallel in a time-division manner. When the GPU 14 completes the data generation process corresponding to a data unit of a first application and is about to switch to a subsequent processing unit, the switching command unit 44 issues an instruction to switch to image generation processing corresponding to a second application different from the first application.

Description

Technical Field

[0001] This invention relates to a data processing technology, and more specifically, to a data processing system, a data processing method, and a computer program. Background Technology

[0002] Servers in the cloud sometimes generate corresponding images in parallel for multiple clients, corresponding to multiple applications (e.g., games) executed by multiple clients, deliver the image generated for each client to the client, and make the image displayed. Summary of the Invention

[0003] [Technical Issues]

[0004] The inventors anticipate that, from an efficiency standpoint, there is room for improvement in servers that generate multiple images corresponding to multiple applications and deliver the generated images to the respective clients.

[0005] The present invention has been designed based on the above-mentioned ideas of the inventors, and one object of the present invention is to provide a technique for making the processing of generating multiple images corresponding to multiple applications or delivering the generated images to the corresponding clients more efficient.

[0006] [Solution to the problem]

[0007] To address the aforementioned problems, one aspect of the present invention provides a data processing system comprising a GPU and an instruction set. The GPU performs time-division multiple data processing steps corresponding to multiple applications executed in parallel. When the GPU completes the processing to generate a data unit corresponding to a first application and to be transferred to a subsequent processing step, the instruction set instructs the GPU to switch to processing to generate data corresponding to a second application different from the first application.

[0008] Another aspect of the present invention is a data processing method. This method is executed by a data processing system comprising a GPU and an instruction set. The data processing method includes a step executed by the GPU, which performs time-division processing to generate multiple data points corresponding to multiple applications executed in parallel; and a step executed by the instruction set, wherein, upon completion of processing by the GPU to generate a data unit corresponding to a first application and to be transferred to a subsequent processing unit, the GPU is instructed to switch to processing to generate an image corresponding to a second application different from the first application.

[0009] It should be noted that any combination of the above-described components and expressions of the present invention, in the form of a device, a program, a recording medium for storing the program, etc., is also valid as a mode of the present invention.

[0010] [Beneficial effects of the invention]

[0011] This invention makes the process of generating multiple images corresponding to multiple applications or delivering the generated images to the corresponding clients more efficient. Attached Figure Description

[0012] Figure 1 This is a diagram illustrating the methods used to execute multiple applications.

[0013] Figure 2 This is a diagram illustrating an example of sharing a GPU across multiple applications.

[0014] Figure 3 This is a diagram illustrating an example of sharing a GPU across multiple applications.

[0015] Figure 4 It is a diagram illustrating the relationship between the timing of compression encoding operations and the required performance.

[0016] Figure 5 This is a block diagram illustrating the functional blocks of a data processing system used in a working example.

[0017] Figure 6 This is a diagram that schematically illustrates the parallel processing of multiple applications in a data processing system.

[0018] Figure 7 This is a diagram illustrating the handshake flow between the CPU and GPU during application processing.

[0019] Figure 8 This is a diagram illustrating the timing of frame buffer switching during application processing.

[0020] Figure 9 This is a diagram illustrating an example of sharing a GPU in a time-sharing manner.

[0021] Figure 10 This is a diagram illustrating an example of sharing a GPU in a time-sharing manner.

[0022] Figure 11 This is a diagram illustrating an example of sharing a GPU in a time-sharing manner.

[0023] Figure 12 This is a diagram illustrating the operation of a compression encoder when processing multiple applications in parallel.

[0024] Figure 13 This means that in Figure 1 The diagram illustrates an example of time-division multiplexing CPU and GPU in System A.

[0025] Figure 14 This means that in Figure 1The diagram illustrates an example of time-division multiplexing CPU and GPU in System A.

[0026] Figure 15 This is a diagram that schematically illustrates the parallel processing of multiple applications in a modified example data processing system.

[0027] Figure 16 This is a diagram that schematically illustrates the parallel processing of multiple applications in a modified example data processing system. Detailed Implementation

[0028] <Background Technology>

[0029] Figure 1 This describes a method for executing multiple applications. Assume two systems are available, System A and System B. System A is equipped with a relatively high-performance CPU (Central Processing Unit) and GPU (Graphics Processing Unit) capable of handling gaming applications (hereinafter referred to as "gaming applications") at up to 60fps (frames per second) with 4K resolution (3840×2160). System B is capable of handling gaming applications at up to 60fps with FHD (Full High Definition) resolution (1920×1080). System A is essentially four times more powerful than System B. Assume System A is backward compatible with System B.

[0030] Here, when multiple games need to be run simultaneously at 60fps with FHD resolution, the first likely option would be to install and use applications such as... Figure 1 The upper part shows multiple systems B. Furthermore, a second possible option would be to process multiple game applications in parallel within a single system A (e.g., processing four games in parallel), such as... Figure 1 As shown in the lower part. This work example relates to the second option and proposes a technique for achieving a physical reduction in the number of installed systems and an improvement in the operational efficiency of each system.

[0031] This work presents a data processing system that enables parallel processing of multiple applications, i.e., simultaneously processing multiple contexts in parallel within the compression encoding section, reducing processing latency and peak performance. An overview of the technical concepts behind this work will be given first.

[0032] <Technical Idea 1 in the Working Example>

[0033] Technical concept 1, as a working example, will provide a description of a method for efficiently processing multiple game applications using a single GPU on a server provided in the cloud (also known as a "cloud server").

[0034] (1) Problem

[0035] Methods for executing multiple applications using a single CPU have become commonplace. For example, it is possible to execute multiple applications in parallel by dividing the CPU into arithmetic units in a spatial direction (also known as "space division"). Alternatively, it is possible to execute multiple applications in parallel by dividing the CPU into arithmetic units in a temporal direction (also known as "time division"). Spatial division of arithmetic units refers to assigning each application to one of multiple CPUs or one of multiple threads on a single CPU. Time division of arithmetic units refers to sharing a single CPU among multiple applications in a time-division manner.

[0036] Here, again in the case of a single GPU handling the processing of multiple applications, a possible option is to divide the arithmetic units spatially or temporally; however, this involves GPU-specific issues. Figure 2 and Figure 3 An example of sharing a single GPU across multiple applications is shown. Figure 2 A and Figure 3 A shows an example of sharing a GPU via time-division, and Figure 2 B and Figure 3 B illustrates an example of sharing a GPU through spatial partitioning. In the diagrams, "CP" in GPU refers to the command processor that processes commands received from the CPU, "SE" refers to the shader engine that performs shader processing, "TU" refers to the texture unit that performs texture processing, "RB" refers to the rendering backend that performs rendering processing, and "L2" and "L3" refer to the level 2 cache and level 3 cache, respectively.

[0037] (1-1) Problems when the spatial GPU is partitioned

[0038] GPUs typically contain multiple arithmetic units that perform shader processing and general-purpose computations in parallel, and a possible option is to physically partition these arithmetic units and allocate one of them to each application for exclusive use. For example, where 2048 pipelines of the shader arithmetic unit are available in the GPU, a possible option would be to allocate 512 pipelines to each of four applications. However, functional blocks that handle tessellation, rasterization, etc., are typically built with the premise that they are performed serially on a single GPU. To partition the GPU in space, these functional blocks actually need to be provided in parallel within a single GPU for this reason, thus leading to increased cost.

[0039] (1-2) Problems when GPUs are partitioned in terms of time

[0040] (1-2-1) Context switching overhead

[0041] Multiple applications use the same arithmetic units one after another. Therefore, it is necessary to drain and read the internal state each time a transition (also known as a “context switch”) occurs. GPUs typically contain multiple arithmetic units (also known as “unified shaders”) in parallel for handling pixel / vertex shader processing and general computations, and these arithmetic units originally had mechanisms to reduce context switching overhead, such as the reuse of internal state storage locations caused by considering preemptive parallel processing of multiple processes in a time-division manner. However, this time-division consideration limits the fixed functionality beyond the unified shader. While transition time periods can be reduced, for example, by reusing registers that hold the internal state, the GPU as a whole has a large amount of internal data, resulting in a non-negligible degradation in chip area efficiency attributable to the increased circuit scale.

[0042] Furthermore, there are no issues even if context information or accessed data is shared between multiple processes that preemptively execute within a single application. However, sharing such information and data between multiple applications is not permitted, and the draining and reading of this information and data must be repeated every time a context switch occurs.

[0043] Furthermore, GPU processing pipelines are longer than CPU processing pipelines, and there is a significant delay from command buffer fetching to the completion of command-based processing. During context switching, it is necessary to wait for all processing in the pipeline to complete after a pause in feeding into the pipeline, which results in an increased amount of time.

[0044] Furthermore, GPUs utilize more fragments of internal state information than CPUs, and none of the various overheads required for context switching are negligible. These overheads include the processing time required to empty and read internal state, the circuitry used for multiplexing to avoid this, and the waiting time until the internal processing in the pipeline is complete.

[0045] (1-2-2) Configuration of game application processing

[0046] In gaming applications, a series of pre-processing steps using a unified shader are bundled together, and the single serialized process that ultimately generates the rendering result in the timed steps of displaying on the screen is executed as a whole. That is, it is possible for gaming applications to exist where time-division execution is impossible, unlike those applications that only handle general-purpose computing (also known as "GPGPUs"), because multiple internal processes are partially highly serialized by means such as waiting for synchronization. Even if context switching overhead can be ignored, there are some cases where viewpoint processing from the processing architecture of a gaming application cannot be handled.

[0047] (2) Solution Technology

[0048] (2-1) When the game application completes its rendering processing to the frame buffer, the granularity of dividing GPU processing into time intervals is set. In the working example's data processing system, the GPU performs processing to generate multiple data entries corresponding to multiple applications executed in parallel in a time-division manner. Then, when the GPU completes the processing to generate a data unit corresponding to the first application and to be handed over to the subsequent processing section, the instruction section of the working example's data processing system (switching instruction section 44, described later) instructs the GPU to switch to processing to generate data corresponding to a second application different from the first application. A data unit is the final product, which will be handed over to the subsequent processing section, and may be, for example, a frame (image) to be stored in the frame buffer.

[0049] In the working example, the granularity of GPU processing in time is set to the timing of frame buffer flips. Typically, each game application ensures two or more frame buffers, and uses two or more frame buffers simultaneously when switching between targets to be drawn and targets to be displayed, and this switching is called "flipping".

[0050] The GPU continues its processing by fetching a command buffer generated by the CPU or other components. After fetching the command associated with drawing a frame of the game application, the GPU stops subsequent fetching, recognizes that its internal processing has finished when the flip occurred, and begins saving the game application's context. If there is ongoing asynchronous computation processing that is out of sync with the frame, the GPU also stops processing and similarly saves the context. The GPU then loads the stored context of the game application to be executed next and resumes execution of that application. This context contains the GPU's internal state and the contents of the command buffer. It should be noted that a single frame buffer can be used where inconsistency between drawing and display processing can be guaranteed. However, issuing a flip notification to signal that the frame buffer is accessible is permissible.

[0051] In GPUs, the amount of internal state and intermediate data that needs to be emptied for context switching immediately after drawing (i.e., when a flip occurs) is reduced compared to when drawing is in progress. This is because, although drawing processing is in progress, a large amount of fine-grained internal state and intermediate data associated with the immediately executed drawing process resides in parallel arithmetic units (e.g., shader pipelines) and fixed functions (e.g., rasterizers). Furthermore, some parts of the drawing process are highly serialized through means such as waiting for synchronization, which is difficult to partition in time. Additionally, there are cases where hidden cancellation intervals with explicit timing are inserted after a flip. In these cases, GPU processing is not required. Therefore, performing a context switch when a flip occurs to reduce and hide overhead is advantageous.

[0052] (2-2) Context switching is performed during the blanking interval of the display timing.

[0053] It should be noted that although the display output device is not usually connected to the cloud gaming server, the timing when the frame buffer is handed over to the compression encoding circuit can be regarded as the blanking interval.

[0054] (2-3) is set to ensure that the GPU has at least the combined computing power required for the corresponding application to be executed in parallel.

[0055] For example, in a game application that generates FHD (1980×1080) images at 60fps, which requires 2 TFLOPS (Taflops per second) of GPU computing power and four such game applications are running in parallel, a GPU with at least 8 TFLOPS of computing power should be used.

[0056] (2-4) Make the GPU run at a frame rate equal to the sum of the expected processing frame rates of the corresponding applications executed in parallel in a time-division manner.

[0057] For example, if four game applications (each generating 60fps) are running in parallel, the GPU frame rate is 240Hz, that is, context switching is performed at 240Hz.

[0058] (2-4-1) Context switching is delayed if a target application, which is executed in parallel in a time-division manner, occupies more of the GPU time slice described in 2-4. The next time slice for the application that has already occupied more than its time slice is skipped, or the next time slice for the application is reduced.

[0059] (2-4-2) Reduce the number of applications executing in parallel to ensure that the GPU is allocated to each application at a sufficient frequency if the GPU usage of a target application executing in parallel in a time-division manner exceeds a large time margin of time stripes or is executed at a high rate. For example, the expected processing load (predicted value of processing load) for each application can be identified in advance. Alternatively, after allocating a GPU usage period to each application, execution can begin, and then the GPU usage period allocated to each of the multiple applications can be examined after measuring the level of excessive processing time (e.g., drawing frame rate count) and the level of latency increase relative to user input.

[0060] While the first system is running, an application that should be reassigned to a different system (the second system) (referred to as a "relocated application") emerges. The relocated application is first activated in the second system, and a context copy from the first system is initiated. Processing of the relocated application continues in the first system until these settings are complete. Then, processing of the relocated application in the first system is paused at a certain time following a context switch.

[0061] (2-5) is configured such that the CPU has at least the combined computing power required by the corresponding application to be executed in parallel in a time-division manner, and operates in sync with the GPU’s rendering frame rate.

[0062] CPU performance can be achieved through a combination of factors such as operating frequency, CPU core count, and thread count. For example, suppose a system runs four game applications in parallel on a cloud server, each using eight single-threaded CPUs running at 1.6 GHz. Alternatively, eight dual-threaded CPUs with the same architecture running at 3.2 GHz could also be used in this system. Furthermore, if each of the four parallel game applications requires 100 GFLOPS (gigaflops per second) of computing power, a CPU with at least 400 GFLOPS of computing power could be used.

[0063] (2-6) Provide a GPU command buffer for each application and switch the target referenced by the GPU when a context switch occurs.

[0064] (2-6-1) can provide another command buffer for computation processing that is asynchronous with the drawing process.

[0065] (2-7) Provide information that indicates each frame in the frame buffer is generated after the others, when it has been generated, and when it should be displayed to the GPU for subsequent processing steps. Subsequent GPU processing steps include, for example, compression encoding, decompression (decoding), communication, and display.

[0066] <Technical Ideas in the Working Example 2>

[0067] Technical concept 2, as a working example, will provide a description of a method for efficiently compressing and encoding multiple images of multiple game applications stored in frame buffers on a cloud server, and for transmitting the compressed and encoded images to a client.

[0068] (1) Problem

[0069] When multiple frame buffers are generated, compressed, and encoded simultaneously, especially when multiple game applications are running on a cloud server and the frame buffers are sent to clients with low latency, sufficient concurrency and computing power are typically required to process the concurrently generated frame buffers in parallel using compression encoders. For example, if four game applications (each generating images at 1920x1080 resolution at 60Hz) are executed in parallel, and four frame buffers are thus completed simultaneously, the system would need four compression encoders capable of compressing and encoding images at 1920×1080 resolution at 60Hz. This can lead to increased system costs.

[0070] Furthermore, if you are attempting to temporarily store multiple images from multiple game applications across multiple frame buffers and compress and send these images with minimal latency, a high-performance compression encoder is required (see description below). Figure 4 (See the right side of the image). Furthermore, if an attempt is made to temporarily store multiple images from multiple game applications in multiple frame buffers and compress and transmit these images with low latency, the interconnects (buses) used to input data to and output data from the compression encoding section are also required to have higher performance. Additionally, peak power consumption increases if multiple images from multiple game applications are temporarily stored in multiple frame buffers and compressed and transmitted with low latency.

[0071] (2) Solution Technology

[0072] The aforementioned problem is addressed by providing the compression encoder with a mechanism for efficiently processing multiple streams and serializing multiple game applications to generate frame buffers. Specific configurations will be described below in sections 2-1 through 2-6.

[0073] (2-1) Serialization timing when multiple game applications terminate their drawing to the frame buffer.

[0074] In multiple game applications running in parallel, for example, the GPU processing start times of the game applications can be shifted relative to each other. Furthermore, timing flips or vertical synchronization timings can be shifted relative to each other in multiple game applications running in parallel. Although the display output device is typically not connected to a cloud gaming server, the timing when the frame buffer is handed over to the compression encoding circuitry can be virtually regarded as vertical synchronization timing. It should be noted that to implement the technical idea of ​​Feature 2, the parallel execution of multiple game applications can be achieved by using multiple GPUs or by spatially or temporally dividing the arithmetic units of a single GPU.

[0075] Figure 4This illustrates the relationship between compression encoding operation timing and required performance. Here, we assume four game applications (Game Application A through Game Application D), each generating an image with a resolution of 1980×1080 at 60fps, and the images from all four applications are stored in a frame buffer. In scheme 70, the images from all four applications are stored in the frame buffer simultaneously. In scheme 70, the number of pixels required to process within four milliseconds is 1920×1080×4. On the other hand, in scheme 72 of the working example, the images from the four game applications are stored in the frame buffer at different timings. Therefore, in scheme 72, the number of pixels required to process within four milliseconds is 1920×1080. Thus, scheme 72 of the working example can achieve compression encoding with low latency while minimizing the required performance of the compression encoder.

[0076] (2-2) The frame buffer is generated at a frame rate equal to the sum of the expected processing frame rates of the multiple applications running in parallel. In other words, multiple images corresponding to the multiple applications are generated at a frame rate equal to the sum of the expected frame rates of the multiple applications running in parallel. For example, in the case of four game applications (each of which should generate images at 60fps) running in parallel, the frame buffer is generated at 240Hz.

[0077] Furthermore, for example, in the case of multiple applications executing in parallel, totaling four games—game A, game B, game C, and game D—frame buffers are generated in the order of game A, game B, game C, game D, game A, game B, game C, game D, etc. However, it should be noted that the order in which frame buffers are generated can be changed if the processing time per frame for each application is variable. In this case, information indicating the application that has generated each frame buffer is stored. It should also be noted that frame buffers generated by each application do not necessarily have to be placed in the same shared memory region, and the compression encoder can refer to multiple memory regions sequentially.

[0078] (2-3) is set such that the compression encoder is able to perform compression encoding at a frame rate equal to the total number of frames of the combination of multiple applications executed in parallel.

[0079] For example, in the case of running four game applications in parallel (each generating an image with a resolution of 1920×1080 at 60fps (60Hz), a compression encoder capable of processing images with a resolution of 1920×1080 generated at 240fps (240Hz) is used.

[0080] (2-4) is configured to cause the compression encoder to sequentially encode a series of frame buffers output from multiple game applications, and then output the encoded data in a partitioned manner for each game.

[0081] In this configuration, the compression encoder is equipped with a feature that identifies which frame buffer belongs to which game. For example, a frame buffer for output images can be provided for each game. Furthermore, when multiple game images are output to a single frame buffer, tags indicating the image's affiliation can be added to the image data. Additionally, data can be transferred to different transmitters on a game-by-game basis.

[0082] (2-5) is configured such that the compression encoder retains past image frames from the data output from the game and processing in a manner different from other games during compression encoding, and improves compression efficiency by referencing this during compression encoding of new images output from the game.

[0083] In the same case, the compression encoder is configured to identify which frame buffer belongs to which game.

[0084] For example, suppose frame images A1 and A2 of game A, frame images B1 and B2 of game B, frame images C1 and C2 of game C, and frame images D1 and D2 of game D are output in the order A1, B1, C1, D1, A2, B2, C2, and D2. In this case, the compression encoder does not refer to A2 used for forward reference during compression and encoding of B2, and instead refers to pre-stored B1 data or data generated during processing during the compression encoding of B1.

[0085] (2-6) is set to enable the compression encoder to perform compression encoding on a unit of partial image (a portion of the applied image) and output the encoded data sequentially on a unit of partial image, ensuring processing with low latency.

[0086] <Details of the system in the working example>

[0087] The following is a description of the configuration used to implement the above technical ideas.

[0088] Figure 5 This is a block diagram illustrating the functional blocks of a data processing system as an example of a working system. Data processing system 10 is a system used as a unit for processing applications, and the cloud server includes one or more data processing systems 10. Data processing system 10 includes a CPU 12, a GPU 14, memory 16, a compression encoder 18 (also referred to as a "video encoder"), and a network interface 20. These components are interconnected via bus 22. GPU 14 and compression encoder 18 are each provided in data processing system 10.

[0089] In the block diagrams of this specification, each element described as a functional block for performing various processing tasks may include, in terms of hardware, circuit blocks, memory, and other LSI (Large-Scale Integration) circuitry, and in terms of software, is implemented by a program loaded into memory, etc. Therefore, those skilled in the art will understand that these functional blocks can be implemented in various ways, such as by hardware alone, by software alone, or by a combination thereof, and are not limited to any of them. For example, a computer program containing a module corresponding to a functional block in CPU 12 or GPU 14 illustrated in the block diagram may be installed into the storage device of the data processing system 10. CPU 12 and GPU 14 can implement the functionality of the respective functional blocks by reading the computer program into memory 16 and executing the computer program.

[0090] Memory 16 stores data referenced or updated by CPU 12, GPU 14, and compression encoder 18. Memory 16 includes command buffer 30 and frame buffer 32. Command buffer 30 stores commands issued by CPU 12 (e.g., commands specifying what to draw) to GPU 14. In the working example, command buffer 30 includes multiple different buffers, each for each game application to be executed in parallel. Frame buffer 32 stores image data generated by GPU 14. In the working example, frame buffer 32 includes multiple different buffers, each for each game application to be executed in parallel. The separate frame buffer for each game application is also referred to as an "application-specific frame buffer."

[0091] Here, the command buffer storing drawing instructions (i.e., instructions for data processing executed synchronously with the switching between applications to be executed by GPU 14) will be referred to as the first command buffer. Command buffer 30 may contain a second command buffer, different from the multiple first command buffers corresponding to multiple game applications, and stores instructions for data processing executed asynchronously with the switching between applications to be executed by GPU 14. The data processing executed asynchronously with the switching between applications can be, for example, general-purpose computational processing other than image processing. GPU 14 can read commands from the second command buffer and execute general-purpose computational processing, regardless of the application to be executed.

[0092] Network interface 20 communicates with the client device according to a predetermined communication protocol. In the operational example, the client device is an information processing device (e.g., a PC, smartphone, or game console) for a user playing the game, displays game images generated by data processing system 10, and receives user commands for the game. Network interface 20 transfers the user's command information sent from the client device to CPU 12. Furthermore, network interface 20 sends image data compressed and encoded by compression encoder 18 to the client device.

[0093] The CPU 12 includes a content determination section 40, a drawing instruction section 42, a switching instruction section 44, and a time holding section 45. The GPU 14 includes a command readout section 50, an image generation section 52, a context storage section 54, and a context readout section 56. The content determination section 40 and the image generation section 52 perform processing dependent on each of the multiple game applications 58 that are executed in parallel. The compression encoder 18 includes an image input section 60 and a compression encoding section 62. Figure 5 At least some of the functional blocks contained in the CPU 12, GPU 14, and compression encoder 18 illustrated herein can be implemented as computer programs. The CPU 12, GPU 14, and compression encoder 18 can implement the functions of their respective functional blocks by executing computer programs.

[0094] Figure 6 The parallel processing of multiple applications in the data processing system 10 is illustrated schematically. CPU 12 performs data processing in parallel for game applications A, B, C, and D, which are executed by multiple client devices. GPU 14 performs processing based on instructions from CPU 12 to generate multiple images corresponding to the multiple game applications executed in parallel in a time-division manner. As previously stated, GPU 14 has a computational power at least equal to the combined sum of the respective GPU computing power required for the multiple game applications executed in parallel.

[0095] Specifically, the content determination section 40 of CPU 12 determines the content of the image to be displayed by each client device (e.g., an image indicating the result of the operation) based on user operation information input from multiple client devices. In other words, the content determination section 40 determines the content of multiple images (e.g., image patterns and coordinate information) corresponding to game application A, game application B, game application C, and game application D.

[0096] The drawing instruction section 42 of CPU 12 stores commands in the command buffer 30 (corresponding to the individual application command buffer of the corresponding game application in the working example) to cause the content of the corresponding game application determined by the content determination section 40 to be drawn.

[0097] The command readout portion 50 of GPU 14 corresponds to Figure 2The CP shown in the figure. Command readout section 50 reads commands that instruct the rendering of an image processed from an individual application command buffer corresponding to the target application, which is being processed at a certain point in time (e.g., game application A, and referred to herein as the "target application"), among multiple game applications being executed in parallel. When the command readout section 50 of GPU 14 receives an application switching instruction (context switching instruction in the working example) from the switching instruction section 44 of CPU 12, and the application to be executed is to be switched, the command readout section 50 switches the command buffer to an individual application command buffer corresponding to the application to be executed next, and reads rendering instructions from the individual application command buffer corresponding to the application that has been executed up to that point.

[0098] The image generation section 52 of GPU 14 corresponds to... Figure 2 SE, TU, and RB are described in the text. The image generation unit 52 generates multiple images corresponding to multiple game applications executed in parallel, and stores the multiple images in the frame buffer 32. Specifically, the image generation unit 52 generates image data for the target application based on the command read out by the command readout unit 50. The image generation unit 52 stores the generated image data for the target application in an individual application frame buffer corresponding to the target application. As described above, the image generation unit 52 generates multiple images corresponding to the multiple game applications at a frame rate equal to the total number of combinations of the expected frame rates of the multiple game applications executed in parallel.

[0099] GPU 14 can provide the following additional information about the image stored in frame buffer 32 to subsequent processing units (e.g., compression encoder 18, communication unit, display unit). The additional information may include (1) data indicating which application the image belongs to, in other words, data indicating from which application the image was generated. Furthermore, the additional information may include (2) data indicating the timing of image generation, and may include, for example, the image number, timestamp, and date information. Additionally, the additional information may include (3) data indicating the timing of image display, and may include, for example, timestamps and date information. This helps ensure that priority allocation, determination of skipped processes, etc., are correctly performed in image processing executed by subsequent processing units. It should be noted that GPU 14 may store the additional information in frame buffer 32 as image metadata via a memory region different from frame buffer 32, or transfer the additional information to subsequent processing units.

[0100] Figure 7 This is a diagram illustrating the handshake flow between the CPU and GPU during application processing. For example... Figure 7As illustrated, when image generation processing is complete, the image generation unit 52 of GPU 14 sends a signal to CPU 12 to notify that drawing has ended. In the working example, the image generation unit 52 generates an image (in other words, a frame) for the target application and stores the image in frame buffer 32. When storage is complete, the image generation unit 52 sends a signal to CPU 12 to notify that drawing has ended.

[0101] Once GPU 14 has completed all the processing required to create a frame corresponding to the first application (e.g., game application A), the switching instruction portion 44 of CPU 12 instructs GPU 14 to switch to processing for generating an image corresponding to a second application (e.g., game application B) different from the first application. The case where all the processing required to create a frame corresponding to the first application (e.g., game application A) is completed can be described as the processing for generating an image corresponding to the first application (e.g., game application A), which is the image to be stored in the frame buffer (also referred to as the final image). Specifically, upon receiving a drawing end notification from GPU 14, the switching instruction portion 44 sends a context switching instruction signal to GPU 14. The context switching instruction may include information specifying a new target application different from the target application up to that point.

[0102] When a context switch instruction is received from CPU 12, the context storage portion 54 of GPU 14 stores the context data of the target application up to that point in memory 16 or a predetermined empty register. As previously described, the context data includes internal state information and intermediate data from the rendering processes of the target application up to that point.

[0103] Furthermore, when a context switch instruction is received from the CPU 12, the context read section 56 of the GPU 14 retrieves the context data of the new target application specified by the context switch instruction from the memory 16 or a predetermined empty register, and sets the retrieved context data in the rendering processing register of the GPU 14. When the context switch is complete, the command read section 50 of the GPU 14 reads a command instructing the image corresponding to the new target application to be drawn from the individual application command buffer corresponding to the new target application and moved to the processing that generates the image corresponding to the new target application.

[0104] The time-holding section 45 of the CPU 12 measures the duration of GPU 14 usage by measuring each of the multiple game applications running in parallel. In other words, the time-holding section 45 measures the time period required for each of the multiple game applications to perform image generation processing.

[0105] In response to receiving a toggle instruction signal from the CPU 12, the image reading section 60 of the compression encoder 18 reads image data from the frame buffer 32 (from multiple individual application frame buffers in the sequence of the working example). The compression encoding section 62 compresses and encodes the image data read by the image reading section 60 according to a predetermined algorithm. When the compression encoding of some image data is completed, the compression encoding section 62 transfers the compressed and encoded image data to the network interface 20 for transmission to the client device.

[0106] In the working example, the switching instruction section 44 of the CPU 12 sends a toggle instruction signal to the compression encoder 18 in response to receiving a drawing end notification from the GPU 14, and simultaneously sends a context switching instruction to the GPU 14. The toggle instruction signal is a signal instructing the switching between multiple frame buffers 32 for a target where image data is written and a target where reference image data (to be read out for compression encoding) is located. In other words, the toggle instruction signal instructs the compression encoder 18 to perform compression encoding of the image stored in the frame buffers 32. The toggle instruction signal may include information specifying a region of the frame buffers 32 where the image undergoing compression encoding is stored (in the working example, a specific individual application frame buffer).

[0107] Figure 8 The diagram illustrates the timing of switching between frame buffers during application processing. This diagram shows an example of dual buffers (here referred to as "buffer 'a' and buffer 'b'"). When GPU 14 writes an image to buffer 'a', compression encoder 18 reads the image stored in buffer 'b' for compression encoding. Conversely, when GPU 14 writes an image to buffer 'b', compression encoder 18 reads the image stored in buffer 'a' for compression encoding. This switching is performed in response to a toggle instruction signal.

[0108] In the data processing system 10, the CPU 12 shifts the timing of the rendering process for each application in the GPU 14 to ensure a predetermined GPU time period allocated to each application for multiple applications to be executed in parallel. Therefore, the image generation section 52 of the GPU 14 generates multiple images corresponding to the multiple applications to be executed in parallel, at different timings for each application, and stores these images in individual application frame buffers, each image corresponding to one of the applications.

[0109] Furthermore, in the data processing system 10, the compression encoder 18 reads multiple images corresponding to multiple applications from various application frame buffers based on a toggle instruction signal sent from the CPU 12. Each application frame buffer corresponds to one of the applications at different timings for each application. This allows for a reduction in the processing time from drawing to compressing data, in other words, a reduction in latency.

[0110] Furthermore, in the data processing system 10, the compression encoder 18 reads multiple images corresponding to multiple applications from the frame buffer 32 at different times for each application, and performs the encoding of the images corresponding to the respective applications as sequential processing. This makes it possible to reduce the processing performance required by the compression encoder 18.

[0111] Figure 9 This illustrates an example of sharing GPU 14 in a time-sharing manner. Figure 9 In A, four systems run in parallel and generate (or display) corresponding images for game applications A, B, C, and D. Figure 9 In B, a single system processes game applications A, B, C, and D in parallel using a time-division multiplexing approach. Figure 9 The system in B is superior in performance to... Figure 9 One of the systems in A is four times higher. Figure 9 The time required for the system in B to process each game application is Figure 9 One-quarter of the system in A (i.e., 1 / 240 of a second).

[0112] Figure 10 It also illustrates an example of sharing GPU 14 in a time-sharing manner. Figure 10 A illustrates the ideal scenario, and the GPU usage time (the sum of image generation time and context switching time) for each of game applications A through D is adapted to within 1 / 240th of a second. Figure 10 In B, on the other hand, the GPU usage time of game application B exceeds 1 / 240 of a second.

[0113] exist Figure 10 In the case shown in B, the data processing system 10 of the working example skips the next GPU allocation for the game application B. That is, if the GPU usage time required to generate an image corresponding to a certain application exceeds a predetermined specified time period, the GPU allocation for the next processing step to generate the image corresponding to the application is skipped.

[0114] For example, the time-holding portion 45 of CPU 12 measures the time period from issuing an instruction to GPU 14 to execute the rendering of a specific game application to receiving a rendering completion notification from GPU 14, or the time period from issuing an instruction to execute the rendering of the next game application, as the GPU usage time period for the specific game application. If the GPU usage time period for the specific game application exceeds a predetermined time period (e.g., in...), Figure 10 In the example of 1 / 240 seconds, the drawing instruction section 42 of CPU 12 skips the drawing instruction for a specific game application once (skips the next instruction).

[0115] As an example of modification, if the GPU usage time required to generate an image corresponding to a certain application exceeds a predetermined allocated time period, the data processing system 10 can reduce the GPU time period for the next processing step of generating the image corresponding to the application. For example, if the GPU usage time period for a specific game application, as measured by the time-holding section 45, exceeds the predetermined time period (in... Figure 10 In the example of 1 / 240 of a second, the drawing instruction section 42 of CPU 12 can reduce the next GPU usage time allocated to a specific game application to less than the time period allocated to it until then. Furthermore, the next GPU usage time allocated to a specific game application can be reduced to less than the normally allocated time period (in...). Figure 10 In the example, this is 1 / 240 of a second, and the time period is determined by the number of applications running in parallel and the frame rate of the respective applications. Furthermore, the GPU usage time allocated to a specific game application can be reduced to be less than the GPU usage time allocated to other game applications.

[0116] It should be noted that skipping processing allocated to game applications that have already occupied GPU 14 for a long period of time, or reducing processing allocated to GPU time periods, can be performed autonomously by GPU 14. In this case, GPU 14 can replace CPU 12 and include a time-holding portion 45 for measuring the GPU usage time period of each game application.

[0117] Figure 11 An example of sharing GPU 14 in a time-division manner is also illustrated. CPU 12 may also include an application allocation section (not shown). The time-holding section 45 of CPU 12 measures the GPU usage time period of each of the multiple game applications sharing GPU 14 in a time-division manner. If the GPU usage time period of a predetermined number or more of the multiple game applications sharing GPU 14 in a time-division manner exceeds a predetermined time period (e.g., 1 / 240 of a second), the application allocation section may reduce the number of game applications sharing GPU 14 (in other words, the number of game applications processed in parallel by data processing system 10).

[0118] Figure 11 Examples are shown where game applications C and D are excluded from the game applications sharing GPU 14 because the GPU usage time of game application A and the GPU usage time of game application B exceed a predetermined time period (e.g., 1 / 240 of a second). That is, as Figure 11 As shown, when four game applications are executed in parallel, and if the number of game applications whose GPU usage time exceeds a predetermined time period reaches 2, the number of game applications executed in parallel can be reduced to 2. The number of game applications whose GPU usage time exceeds the predetermined time period serves as a threshold for reducing the number of game applications executed in parallel, and the reduction in the number of game applications when the threshold is exceeded can be determined experimentally using the data processing system 10.

[0119] It should be noted that when the number of game applications running in parallel on its own system decreases, the application allocation portion of CPU 12 can coordinate with the application allocation portion of another data processing system 10 to reallocate the processing of at least one game application running in parallel on its own system to the other data processing system 10. For example, if a predetermined number or more of the multiple applications running in parallel on its own system have a GPU occupancy time exceeding a predetermined time period (e.g., 1 / 240 of a second), the application allocation portion of CPU 12 can reallocate at least one of the multiple applications to another data processing system.

[0120] Specifically, if the processing time period (GPU occupancy time period) of a particular application exceeds a predetermined time period in a micro-time unit (i.e., an extremely small time unit, such as 1 / 240 of a second), the data processing system 10 skips the next GPU allocation for the application (skips). Figure 10 (Application B is processed in this context). Simultaneously, if an unacceptable excess (e.g., exceeding a predetermined time period or count) occurs within the time period of interest, and the statistics of the allocation results have already been performed within a macroscopic time unit (i.e., a sufficiently long time unit such as 10 seconds), the data processing system 10 reallocates some of the multiple applications executing in parallel to another data processing system. It should be noted that applications with large excess times or counts may be preferentially selected as applications to be reallocated to another data processing system.

[0121] Here, the application to be reassigned will be referred to as the target application, the data processing system 10 that has already executed the target application will be referred to as the first system, and the data processing system 10 to which the target application will be reassigned will be referred to as the second system. The application allocation part of the CPU 12 of the first system can coordinate with the application allocation part of the CPU 12 of the second system to perform the following processing tasks (1) to (4) listed below.

[0122] (1) The application allocation section of the CPU 12 of the first system notifies the application allocation section of the CPU 12 of the second system of the identification information of the target application. The processing section on the second system side (e.g., CPU 12, GPU 14) loads the program of the target application from the memory 16 based on the identification information of the target application and starts the target application.

[0123] (2) The application allocation section of the CPU 12 of the first system sends context data about the target application stored in the first system to the second system. The application allocation section of the CPU 12 of the second system sets the context data in the processing section (e.g., CPU 12, GPU 14).

[0124] (3) The target application runs on both the first system and the second system until a context switch of the target application is performed in the first system at a specific time after the completion of the task (1) described above. The application allocation part of the CPU 12 of the first system stops the execution of the target application in the first system in response to the context switch triggered at the specific time mentioned above, and notifies the application allocation part of the CPU 12 of the second system of this effect. The application allocation part of the CPU 12 of the first system sends the context data of the target application of the first system stored by the context switch to the second system at the specific time mentioned above, and the application allocation part of the CPU 12 of the second system sets the context data in the processing part (e.g., CPU 12, GPU 14).

[0125] (4) When the execution of the target application in the first system is notified to stop and the context data of the target application is set up, the application allocation part of the CPU 12 of the second system begins the execution of the target application through each processing part of the second system. It should be noted that the context setting and execution start of the target application in the second system are expected to be performed within the processing time interval (1 / 60 second in this example) of the final context switch allocation of the target application from the first system.

[0126] The specific timing in the task (3) above can be selected under the expectation that the storage of the context of the first system, the transfer of the context from the first system to the second system, and the setting of the context in the second system will be completed within the allocated processing time interval. For example, the specific timing can be determined after making a decision about whether the transmission unit provided between the first system and the second system is occupied by other processing.

[0127] Therefore, after taking over the latest context of the target application in the first system, the second system resumes the execution of the target application without any time lag or user-identified transition of the running system.

[0128] The context can be repeatedly copied from the first system to the second system and its settings until the specific timeout mentioned above arrives. For example, if the transfer, setting, and restoration of all context processing cannot be completed within a single allocated processing time interval due to a large amount of context data, the transfer, setting, and restoration of processing can be performed in multiple separate instances. Furthermore, if it is expected that the storage of the context in the first system, the transfer of the context from the first system to the second system, and the setting of the context in the second system will be completed within the allocated processing time interval, the storage, transfer, and setting of the context may not actually be completed in time. The application executed by the first system can continue, and the copying and setting of the context can be performed again later.

[0129] Furthermore, the data processing system 10 is configured such that the compression encoder 18 reads multiple images corresponding to multiple applications from the frame buffer 32 at different timings for each application. In this working example, the drawing instruction section 42 of the CPU 12 writes commands instructing drawing to the command buffer 30 at different timings for each of the parallel-executing applications. Therefore, the GPU 14 generates images at different timings for each of the parallel-executing applications and writes the images to the frame buffer 32 (individual application frame buffer) at different timings for each of the parallel-executing applications. Additionally, the switching instruction section 44 of the CPU 12 notifies the compression encoder 18 of toggle instructions at different timings for each of the parallel-executing applications.

[0130] Figure 12 This diagram illustrates the operation of the compression encoder 18 in the context of parallel processing of multiple applications. It includes a timing diagram illustrating the timing when the compression encoder 18 reads an image from the frame buffer 32 (scanning) and the timing when the compression encoder 18 compresses and encodes the read image (encoding).

[0131] In the working example, the switching instruction section 44 of CPU 12 sends toggle instruction signals to the compression encoder 18 at different timings for each game application. Therefore, as... Figure 4 (Option 72) and Figure 12 As shown, the image reading section 60 of the compression encoder 18 reads multiple images corresponding to multiple game applications from the frame buffer 32 (specifically, the individual application frame buffer corresponding to the game application to be processed) at different timings for each game application.

[0132] Moreover, such as Figure 4 (Option 72) and Figure 12 As shown, the compression encoding section 62 of the compression encoder 18 compresses and encodes multiple images corresponding to multiple game applications at different timings for each game application. The compression encoding section 62 performs the compression and encoding of the multiple images corresponding to the multiple game applications as sequential processing. Moreover, as previously described, the compression encoding section 62 has the performance to encode images at a frame rate equal to the combined sum of the expected frame rates of the multiple game applications executed in parallel.

[0133] The compression encoding portion 62 of the compression encoder 18 stores past images of each of a plurality of game applications running in parallel in a predetermined storage area, such that the past images are distinguishable from each other for each game application. In other words, the compression encoding portion 62 stores past images corresponding to a game application in the predetermined storage area in a manner that distinguishes them from images corresponding to another game application. When compressing and encoding a new image corresponding to a particular game application, the compression encoding portion 62 references the past images of the game applications stored in the aforementioned storage area. Therefore, the efficiency of compressing the corresponding images of the plurality of game applications running in parallel is enhanced.

[0134] The following is a description of an example of CPU and GPU allocation.

[0135] Figure 1 Each of the systems B in the system comprises eight CPUs, each capable of processing one thread by running at 1.6 GHz. Meanwhile, Figure 1 System A in the example consists of eight CPUs, each capable of processing two threads by running at 3.2 GHz. Therefore, System A has roughly four times the CPU performance of System B. The ability of System A's CPUs to process two threads refers to the fact that the CPU hardware has mechanisms to support thread processing, allowing a single CPU to execute two applications simultaneously.

[0136] Figure 13 and 14 It shows in Figure 1The example below illustrates a time-sharing allocation of CPU and GPU in System A. The four applications (Application A through Application D) executed here were developed with eight CPUs (each running at 1.6 GHz) available. Applications are executed by assigning each application to a thread on each of the eight CPUs. Assuming two threads are supported as a single CPU hardware unit and the four applications will execute in parallel as in System A, the four threads are executed in parallel by providing two software control threads within a single hardware thread.

[0137] Here, CPU thread processing refers to the state in which an application is processed on the CPU at a much smaller granularity (e.g., 1 / 240 of a second) than the predetermined time period allocated to the GPU, regardless of hardware or software control. A state can be created in which each application runs by essentially occupying the CPU of interest.

[0138] Applications running on each CPU thread need to detect changes occurring outside the CPU in real time and take necessary actions. Furthermore, to coordinate with external systems in real time, similar to System B, applications running on their respective CPU threads continue to run in parallel on the CPU, regardless of the predetermined time slot allocated to the GPU or the actual time slot occupied by the application on the GPU. This allows each application running on each CPU thread to run synchronously with the processing of each application running on the GPU without latency.

[0139] As in Figure 13 and 14 As shown, when the CPU is allocated to each application, CPU usage is allocated regardless of the predetermined time period allocated to the GPU. That is, the CPU 12 of the data processing system 10 executes multiple applications in parallel, regardless of the time period allocated to each application in the GPU 14, so that the resources of the CPU 12 are occupied in parallel by the respective applications. This allows applications on the CPU 12 to be processed synchronously with external executions including the GPU 14, while maintaining real-time performance.

[0140] Furthermore, in situations where the number of CPUs is insufficient to handle the number of applications running in parallel, such as... Figure 1 and Figure 13 and 14 As shown in System A, the CPU executes multiple applications to be run in parallel by using threads controlled by hardware or software.

[0141] Furthermore, as previously described, a GPU command buffer is made available for each of the applications to be executed in parallel. The CPU stores data processing instructions (e.g., drawing instructions) for multiple applications to be executed in parallel in the command buffer corresponding to the respective application. The GPU reads out the data processing instructions (e.g., drawing instructions) for the multiple applications to be executed in parallel from the command buffer and performs the data processing (e.g., drawing processing) corresponding to each application. This makes it possible to continue processing for a particular application on the CPU and to write instructions addressed to the GPU to the command buffer corresponding to the particular application at any time, even if processing for the particular application is not allocated to the GPU (and the GPU is not occupied by the particular application).

[0142] The invention has been described above based on working examples. Those skilled in the art will understand that these working examples are illustrative and that the combination of constituent elements and processes can be modified in various ways, and such modifications also fall within the scope of the invention.

[0143] A description of the first modified example will be given. The data processing system 10 can be connected to a display device that displays an image generated by the data processing system 10. The switching instruction section 44 of the CPU 12 can store data indicating the blanking interval (in other words, the interval during which no video image is displayed) while an image is displayed on the display device. The switching instruction section 44 can send a context switching instruction to the GPU 14 and a toggle instruction signal to the compression encoder 18 during the blanking interval while the image is displayed on the display device. As previously mentioned, GPU processing is not required during the blanking interval. Therefore, the impact of context switching processing on image generation processing (e.g., latency in image processing) can be minimized.

[0144] A description of a second modified example will be given. Although the context switching performed by GPU 14 has been described in the working example above, the context switching can also be performed by compression encoder 18. For example, compression encoder 18 may include (as GPU 14) a context readout section that reads the context about a particular application from memory 16 (e.g., the internal state of compression encoder 18 or previously processed data) and sets the context in compression encoder 18 before initiating compression and encoding of an image corresponding to the application. Moreover, compression encoder 18 may include a context storage section that saves the context about the particular application from compression encoder 18 to memory 16 when the compression encoding of the image corresponding to the application is completed.

[0145] A description of a third modification example will be given below. As proposed in the applicant's patent application (Patent Application No. 2010-016144), the data processing system 10 may include at least one of a hardware function block designed solely to perform context switching of the GPU 14 and a hardware function block designed solely to perform context switching of the compression encoder 18. This hardware function block is independent of the CPU 12, GPU 14, memory 16, and compression encoder 18, and is also referred to hereinafter as a "context switching engine".

[0146] The context switching engine accesses memory 16 via DMA (Direct Memory Access). When notified of an application switching timing or detected by itself, the context switching engine (1) selects the context of each application stored in memory 16, passes the context to the processing unit (e.g., GPU 14, compression encoder 18) without software mediation, and sets the context. Alternatively, the context switching engine (2) transfers and saves the context held in the processing unit (e.g., GPU 14, compression encoder 18) to memory without software mediation.

[0147] Specifically, when GPU 14 completes the processing to generate an image corresponding to the first application and begins processing to generate an image corresponding to the second application, the context switching engine that performs the context switching of GPU 14 can detect the application switching timing based on notifications from the switching instruction section 44 of CPU 12, notifications from GPU 14, and timing indicated by video signals. Then, the context switching engine can save the context of the first application from GPU 14 to memory 16 and set the context of the second application from memory 16 to GPU 14, without needing to handle software that manages multiple applications executing in parallel.

[0148] Furthermore, when the compression encoder 18 completes encoding of the image corresponding to the first application and initiates encoding of the image corresponding to the second application, the context switching engine that performs the context switching of the compression encoder 18 can detect the application switching timing based on notifications from the switching instruction section 44 of the CPU 12, notifications from the compression encoder 18, timing indicated by the video signal, etc. Then, the context switching engine can save the context of the first application from the compression encoder 18 to the memory 16, and set the context of the second application from the memory 16 to the compression encoder 18, without handling software that manages multiple applications executing in parallel. The software that manages multiple applications is typically software executed by the CPU 12, and can be, for example, software that manages the execution order of applications or resource allocation.

[0149] A description of the fourth modified example will be given. Figure 15 and Figure 16The diagram schematically illustrates parallel processing of multiple applications in a modified example data processing system 10. The data processing system 10 may have a display pipeline 64 (in other words, a video pipeline) between a GPU 14 and a compression encoder 18.

[0150] Display pipeline 64 can perform time-division multiplexing of images supplied from other blocks that act as image providers (e.g., GPU 14 or a video decoder not shown) (in other words, images from multiple applications to be processed in parallel).

[0151] exist Figure 16 In the process, (1) GPU 14 performs rendering processing for two applications (application A and application B) in a time-division manner, generates two final images for each application, and stores the images in frame buffer 32. (2) Display pipeline 64 scales and merges the two final images for each application. (3) Compression encoder 18 compresses and encodes the images that have been scaled and merged for each application.

[0152] A description of the fifth modified example will be given. Although the switching instruction section 44 is provided in CPU 12 in the working example above, the switching instruction section 44 can be provided in a function block other than CPU 12 (e.g., GPU 14).

[0153] The technical ideas described in the working example are also applicable to situations other than where multiple game applications are executed in parallel. For example, the technical ideas are also applicable to situations where other types of applications (e.g., office software or business system programs) replace or run alongside game applications.

[0154] Any combination of the above working examples and modified examples is also effective for embodiments of this disclosure. New embodiments resulting from such combinations have the advantageous effects of combining the working examples and modified examples. Those skilled in the art will also understand that the functionality delivered by each of the components recited in the claims is implemented by each of the constituent elements indicated in the working examples and by modified examples, individually or in coordination with each other.

[0155] The techniques described in the working examples and modified examples can be identified by the following patterns.

[0156] [Feature 1-1]

[0157] A GPU, adapted to perform time-division multiple data processing operations corresponding to multiple applications running in parallel, wherein the GPU is a graphics processing unit; and

[0158] The instruction section is adapted to instruct the GPU to switch to generating data corresponding to a second application, different from the first application, when the GPU has completed the processing of generating a data unit corresponding to the first application and which will be handed over to the subsequent processing section.

[0159] According to this data processing system, it is possible to effectively switch to processing that generates data corresponding to a second application, because the amount of data for the first application that needs to be saved is immediately reduced after processing to generate a data unit corresponding to the first application is completed. In other words, it is possible to effectively generate multiple data entries corresponding to multiple applications, while reducing the amount of hardware resources and improving system availability by using a single GPU to perform the processing of generating multiple data entries corresponding to multiple applications.

[0160] [Features 1-2]

[0161] According to the data processing system of feature 1-1, the data unit to be transferred to the subsequent processing part is an image.

[0162] According to this data processing system, it is possible to effectively switch to the processing of generating an image corresponding to the second application, because the amount of data about the first application that needs to be saved is immediately reduced after the processing of generating the image corresponding to the first application is completed.

[0163] [Features 1-3]

[0164] According to the data processing system of feature 1-1 or 1-2, wherein, when the processing of generating the data unit corresponding to the first application is completed in the GPU, context data about the first application is stored in a predetermined storage area.

[0165] This data processing system reduces the amount of context data that needs to be saved for the first application and enables efficient context switching. In other words, by performing context switching after GPU rendering is complete, interruption and switching processing within the GPU is easier, thus reducing the processing space required for context switching.

[0166] [Features 1-4]

[0167] The data processing system as described in claim 1 or 2 further includes:

[0168] The conversion section is adapted to, when the GPU completes the processing of generating the data unit corresponding to the first application and initiates the processing of generating data corresponding to the second application, save context data about the first application from the GPU to a predetermined storage area, and set context data about the second application from the storage area to the GPU, without the need for an intermediary software processing that manages the multiple applications.

[0169] According to this data processing system, in addition to the beneficial effects similar to those of features 1-3, it also achieves the beneficial effect of completing context switching even faster.

[0170] [Features 1-5]

[0171] According to any one of features 1-1 to 1-4, in a data processing system, if the GPU occupancy time required for the processing to generate the data unit corresponding to the first application exceeds a predetermined allocated time period, the time period allocated to the next processing to generate the data unit corresponding to the first application is reduced, or the GPU allocation for the next generation processing is skipped.

[0172] According to this data processing system, excessive impacts (e.g., latency) on the GPU usage time of the first application in data processing for other applications on the GPU can be reduced, while maintaining fairness among applications executing in parallel. That is, even if no context switching is performed within each predetermined allocated time period at a microscopic level such as 1 / 240th of a second, each application executes at a rate appropriate to the predetermined allocated time periods at a macroscopic level (e.g., from seconds to minutes). Therefore, it can be ensured that the behavior (including the realism of processing) of each application in the data processing system is consistent with that of the application as... Figure 1 The individual system in system B performs the same behavior.

[0173] [Features 1-6]

[0174] According to any one of features 1-1 to 1-5, in a data processing system, when a predetermined number or more applications in the plurality of applications have a GPU occupancy time exceeding a predetermined allocated time period, the number of applications that generate the one data unit by the GPU is reduced.

[0175] This data processing system can guarantee the GPU usage time required for application data generation and processing.

[0176] [Features 1-7]

[0177] According to any one of features 1-1 to 1-5, in a data processing system, if the GPU occupancy time of a predetermined number or more applications in the plurality of applications exceeds a predetermined allocation time period, at least one of the plurality of applications is allocated to another data processing system.

[0178] This data processing system can guarantee the GPU usage time required for application data generation and processing.

[0179] [Features 1-8]

[0180] According to the data processing system of features 1-7, wherein, when the at least one application is assigned to the other data processing system, (1) the application is started in the other data processing system, (2) context data about the application stored in the self-system is sent to the other data processing system, and (3) in response to a context switch at a specific time of the application in the self-system as a trigger, the processing of the application in the self-system is stopped.

[0181] Based on this data processing system, another system can appropriately take over the processing of applications running on its own system.

[0182] [Features 1-9]

[0183] The data processing system according to any one of features 1-1 to 1-8 further includes:

[0184] The encoding part, in which

[0185] The GPU generates an image as a data unit and stores the generated image in a frame buffer.

[0186] When the encoding section is instructed to encode the image stored in the frame buffer, the instruction section instructs the GPU to perform a switch.

[0187] According to the data processing system, when the processing of generating a unit of data corresponding to one application is completed, it reliably instructs a switch to be executed to generate data corresponding to another application.

[0188] [Features 1-10]

[0189] According to the data processing system of features 1-9, the GPU provides the subsequent processing unit with at least one of the following multiple pieces of data about the image stored in the frame buffer: (1) data indicating which application the image belongs to, (2) data indicating when the image was generated, and (3) data indicating when the image is displayed.

[0190] Based on this data processing system, it is possible to correctly operate subsequent processing sections (e.g., compression / decompression section, communication section, display section).

[0191] [Features 1-11]

[0192] The data processing system according to any one of features 1-1 to 1-10, wherein the instruction portion instructs the switching to be performed during the blanking interval while displaying an image.

[0193] According to this data processing system, no GPU processing is required during the blanking interval. Therefore, the impact of context switching processing on image generation processing (e.g., latency in image processing) can be avoided.

[0194] [Features 1-12]

[0195] According to any one of features 1-1 to 11-11, the data processing system wherein the GPU has at least a computing power equal to the combined sum of the respective GPU computing powers required by the plurality of applications.

[0196] According to this data processing system, it is possible to keep the latency of generating multiple data points corresponding to multiple applications to a minimum. For example, when the data generated by the GPU is images, the latency of displaying multiple images corresponding to multiple applications can be kept to a minimum.

[0197] [Features 1-13]

[0198] The data processing system according to any one of features 1-1 to 1-12 further includes:

[0199] A CPU, adapted to execute the multiple applications in parallel, wherein the CPU is a central processing unit.

[0200] The CPU has a computing power that is at least equal to the sum of the corresponding CPU computing capabilities required by the plurality of applications.

[0201] This data processing system can minimize the latency of processing multiple applications that are executed in parallel.

[0202] [Features 1-14]

[0203] According to the data processing system described in features 1-13, the CPU executes the multiple applications in parallel, with the CPU resources being occupied in parallel by the respective applications, regardless of the time period allocated to each application in the GPU.

[0204] According to this data processing system, the CPU can perform synchronous processing with external functional blocks, including the GPU, while maintaining real-time operation.

[0205] [Features 1-15]

[0206] The data processing system according to features 1-14 further includes:

[0207] The command buffers corresponding to the plurality of applications, wherein

[0208] The CPU stores data processing instructions for the plurality of applications in the command buffer corresponding to the respective application, and

[0209] The GPU reads the data processing instructions for the plurality of applications from the command buffer.

[0210] According to this data processing system, application-specific processing can continue on the CPU even when the processing of a specific application is not allocated to the GPU.

[0211] [Features 1-16]

[0212] The data processing system according to any one of features 1-1 to 1-12 further includes:

[0213] Multiple command buffers, corresponding to the multiple applications, and adapted to store data processing instructions regarding the multiple applications, wherein,

[0214] When switching applications to be executed, the GPU switches between command buffers from which it reads the data processing instructions.

[0215] According to this data processing system, the processing of an application can be executed in a function block that issues data processing instructions asynchronously with the processing of the application in the GPU.

[0216] [Features 1-17]

[0217] The data processing system based on features 1-16 also includes:

[0218] Unlike the multiple command buffers, this command buffer is adapted to store instructions for data processing that can be executed asynchronously with switching between applications to be executed by the GPU.

[0219] According to this data processing system, the switching between the GPU and the application to be executed by the GPU can be asynchronous, and data processing (e.g., general computing processing, such as numerical computation) can be executed asynchronously with the switching.

[0220] [Features 1-18]

[0221] A data processing method executed by a data processing system, the data processing system including a GPU and an instruction set, wherein the GPU is a graphics processing unit, the data processing method comprising:

[0222] The steps executed by the GPU involve time-division processing to generate multiple data entries corresponding to multiple applications running in parallel; and

[0223] The step executed by the instruction section, upon the GPU completing the processing of generating a data unit corresponding to the first application and to be handed over to the subsequent processing section, instructs the GPU to switch to processing of generating an image corresponding to a second application different from the first application.

[0224] According to this data processing method, it is possible to effectively switch to processing that generates data corresponding to the second application, because the amount of data about the first application that needs to be saved is immediately reduced after processing to generate a data unit corresponding to the first application is completed. In other words, it is possible to effectively generate multiple data entries corresponding to multiple applications, while reducing the amount of hardware resources and improving system availability by using a single GPU to perform the processing of generating multiple data entries corresponding to multiple applications.

[0225] [Features 1-19]

[0226] A computer program for causing a data processing system to perform the following steps, the data processing system including a GPU and an instruction set, the GPU being a graphics processing unit:

[0227] The steps executed by the GPU involve time-division processing to generate multiple data entries corresponding to multiple applications running in parallel; and

[0228] The step executed by the instruction section, upon the GPU completing the processing of generating a data unit corresponding to the first application and to be handed over to the subsequent processing section, instructs the GPU to switch to processing of generating an image corresponding to a second application different from the first application.

[0229] According to this computer program, it is possible to efficiently switch to processing that generates data corresponding to a second application, because the amount of data for the first application that needs to be saved is immediately reduced after processing to generate a data unit corresponding to the first application is completed. In other words, it is possible to efficiently generate multiple data entries corresponding to multiple applications, while reducing the amount of hardware resources and improving system availability by using a single GPU to perform the processing of generating multiple data entries corresponding to multiple applications.

[0230] [Feature 2-1]

[0231] A data processing system, comprising:

[0232] The image generation section is adapted to generate multiple images corresponding to multiple applications executed in parallel, and to store the multiple images in a frame buffer; and

[0233] The encoding section is adapted to read the image stored in the frame buffer and encode the image, wherein...

[0234] The encoding section reads multiple images corresponding to multiple applications from the frame buffer at different timings for each application.

[0235] According to this data processing system, it is possible to achieve encoding with low latency and keep the required performance of the encoder to a minimum.

[0236] [Feature 2-2]

[0237] According to the data processing system of feature 2-1, the encoding portion performs sequential processing of the encoding of the plurality of images corresponding to the plurality of applications.

[0238] According to this data processing system, it is possible to achieve encoding with low latency and keep the required performance of the encoder to a minimum.

[0239] [Features 2-3]

[0240] According to the data processing system described in feature 2-1 or 2-2, the image generation section generates the plurality of images corresponding to the plurality of applications at different timings for each application.

[0241] This data processing system can reduce the performance required for image generation (e.g., GPU).

[0242] [Features 2-4]

[0243] The data processing system according to any one of features 2-1 to 2-3 further includes:

[0244] CPU, among which

[0245] When the rendering process for each application is initiated via the image generation section by allocating predetermined time periods to each application, CPU offset timing, and

[0246] The encoding section reads multiple images corresponding to multiple applications from the frame buffer at different timings for each application.

[0247] Based on this data processing system, it is possible to reduce the required performance of the image generation part (e.g., GPU) and the required performance of the encoding part.

[0248] [Features 2-5]

[0249] The data processing system according to feature 2-1 further includes:

[0250] CPU, among which

[0251] When the rendering process for each application is initiated via the image generation section by allocating predetermined time periods to each application, CPU offset timing, and

[0252] The encoding section reads multiple images corresponding to multiple applications from the frame buffer at different timings for each application, and performs the encoding of the multiple images corresponding to multiple applications as a sequential process.

[0253] Based on this data processing system, it is possible to reduce the required performance of the image generation part (e.g., GPU) and the required performance of the encoding part.

[0254] [Features 2-6]

[0255] According to any one of features 2-1 to 2-5, the data processing system wherein the image generation section generates the plurality of images at a frame rate equal to the sum of the expected frame rates of the plurality of applications.

[0256] According to this data processing system, it is possible to keep the latency to a minimum when generating multiple images corresponding to multiple applications, and to keep the latency to a minimum when displaying multiple images corresponding to multiple applications.

[0257] [Features 2-7]

[0258] According to any one of features 2-1 to 2-6, the data processing system wherein the encoding portion has the capability to encode an image at a frame rate equal to the sum of the expected frame rates of the plurality of applications.

[0259] According to this data processing system, the latency can be kept to a minimum when encoding multiple images corresponding to multiple applications, and the latency can also be kept to a minimum when displaying multiple images corresponding to multiple applications.

[0260] [Features 2-8]

[0261] According to any one of features 2-1 to 2-7, the data processing system wherein the encoding portion stores past images corresponding to a specific application in a manner distinguishable from images corresponding to other applications, and when compressing and encoding a new image corresponding to the specific application, references are made to the past images corresponding to the application.

[0262] This data processing system can increase the compression efficiency of images from multiple applications that execute at different times.

[0263] [Features 2-9]

[0264] The data processing system according to any one of features 2-1 to 2-8 further includes:

[0265] The conversion section is adapted to save the context of the first application from the encoding section to the storage section and set the context of the second application from the storage section to the encoding section when the encoding section completes the encoding of the image corresponding to the first application and initiates the encoding of the image corresponding to the second application, without the need for an intermediary that manages the software processing of the multiple applications.

[0266] According to this data processing system, context switching can be performed at high speed in the encoding section, and the processing latency of the encoding section can be kept to a minimum, for example.

[0267] [Features 2-10]

[0268] A data processing method executed by a data processing system, the data processing system including an image generation part and an encoding part, the data processing method comprising:

[0269] The steps performed by the image generation section are for generating multiple images corresponding to multiple applications executed in parallel, and storing the multiple images in a frame buffer; and

[0270] The steps performed by the encoding section are used to read multiple images corresponding to multiple applications from the frame buffer at different timings for each application, and to encode the read images.

[0271] According to this data processing method, it is possible to achieve encoding with low latency while keeping the encoder's required performance to a minimum.

[0272] [Feature 2-11]

[0273] A computer program for causing a data processing system to execute, the data processing system including an image generation section and an encoding section, to perform:

[0274] The steps performed by the image generation section are for generating multiple images corresponding to multiple applications executed in parallel, and storing the multiple images in a frame buffer; and

[0275] The steps performed by the encoding section are used to read multiple images corresponding to multiple applications from the frame buffer at different timings for each application, and to encode the read images.

[0276] According to this computer program, it is possible to achieve encoding with low latency and keep the required performance of the encoder to a minimum.

[0277] [Industrial Applicability]

[0278] This invention is applicable to systems that execute multiple applications in parallel.

[0279] [List of Reference Symbols]

[0280] 10 Data Processing System

[0281] 12 CPU

[0282] 14 GPU

[0283] 18 Compression Encoder

[0284] 32-frame buffer

[0285] 42 Drawing Instructions

[0286] 44 Switching Instruction Section

[0287] 52 Image Generation Section

[0288] 54 Context storage section

[0289] 56. Context Reading Section

[0290] 62. Compression Encoding Section

Claims

1. A data processing system comprising: a GPU adapted to perform processing of generating a plurality of pieces of data corresponding to a plurality of applications executed in parallel in a time-division manner, the GPU being a graphics processing unit; and an instruction section adapted to instruct the GPU to switch to processing of generating data corresponding to a second application different from a first application in a case where the GPU completes processing of generating one data unit corresponding to the first application and to be handed over to a subsequent processing section; wherein, in a case where a GPU occupancy time period of a predetermined number or more of the plurality of applications exceeds a predetermined allocation time period, at least one of the plurality of applications is allocated to another data processing system, and in a case where the at least one application is allocated to the another data processing system, (1) the application is started in the another data processing system, (2) context data about the application stored in a self system is transmitted to the another data processing system, and (3) processing of the application in the self system is stopped in response to a context switch at a certain timing of the application in the self system as a trigger. The one data unit to be handed over to the subsequent processing section is an image.

2. The data processing system of claim 1, wherein, In a case where the processing of generating the one data unit corresponding to the first application is completed in the GPU, context data about the first application is stored in a predetermined storage area.

3. The data processing system of claim 1, wherein, 4. The data processing system according to claim 1, further comprising: a conversion section adapted to save context data about the first application from the GPU to a predetermined storage area and to set context data about the second application from the storage area to the GPU in a case where the GPU completes the processing of generating the one data unit corresponding to the first application and initiates the processing of generating data corresponding to the second application, without an intermediary of software processing that manages the plurality of applications. In a case where a GPU occupancy time period required for the processing of generating the one data unit corresponding to the first application exceeds a predetermined allocation time period, a time period allocated to next processing of generating the one data unit corresponding to the first application is reduced, or GPU allocation to next generation processing is skipped.

5. The data processing system of claim 1, wherein, In a case where a GPU occupancy time period of a predetermined number or more of the plurality of applications exceeds a predetermined allocation time period, the number of applications of the one data unit generated by the GPU is reduced.

6. The data processing system of claim 1, wherein, 7. The data processing system according to claim 1, further comprising: an encoding section, wherein the GPU generates an image as the one data unit and stores the generated image in a frame buffer, and the instruction section instructs the GPU to perform switching when instructing the encoding section to encode the image stored in the frame buffer. ​ 8. The data processing system of claim 7, wherein, The GPU provides to the subsequent processing section at least one of the following pieces of data regarding the image stored in the frame buffer: (1) data indicating which application the image is of, (2) data indicating when the image was generated, and (3) data indicating when the image is to be displayed.

9. The data processing system of claim 1, wherein, The instruction section instructs the switching to be performed during a blanking interval while an image is being displayed.

10. The data processing system of claim 1, wherein, The GPU has at least a computing power equal to the combined sum of the respective GPU computing powers required by the plurality of applications.

11. The data processing system of claim 1, further comprising: a CPU adapted to execute the plurality of applications in parallel, the CPU being a central processing unit, wherein the CPU has at least a computing power equal to the combined sum of the respective CPU computing powers required by the plurality of applications.

12. The data processing system of claim 11, wherein, The CPU executes the plurality of applications in parallel in a manner that the resources of the CPU are occupied by the respective applications in parallel, regardless of the time period allocated to each application in the GPU.

13. The data processing system of claim 12, further comprising: a command buffer corresponding to the plurality of applications, wherein the CPU stores data processing instructions regarding the plurality of applications in the command buffer corresponding to the respective application, and the GPU reads out the data processing instructions regarding the plurality of applications from the command buffer.

14. The data processing system of claim 1, further comprising: a plurality of command buffers corresponding to the plurality of applications and adapted to store data processing instructions regarding the plurality of applications, wherein, in the case of switching the application to be executed, the GPU switches between the command buffers from which the data processing instructions are read out.

15. The data processing system of claim 14, further comprising: a command buffer different from the plurality of command buffers and adapted to store instructions for data processing, the instructions being capable of being executed asynchronously with the switching between applications to be executed by the GPU.

16. A data processing method performed by a data processing system, the data processing system comprising a GPU and an instruction section, the GPU being a graphics processing unit, the data processing method comprising: a step performed by the GPU to perform processing to generate a plurality of pieces of data corresponding to a plurality of applications executed in parallel in a time-division manner; and a step performed by the instruction section to instruct the GPU to switch to processing to generate an image corresponding to a second application different from a first application in the case where the GPU completes processing to generate one piece of data corresponding to the first application and to be handed over to a subsequent processing section; wherein, in a case where a GPU occupancy time period of a predetermined number or more of the plurality of applications exceeds a predetermined allocation time period, at least one application of the plurality of applications is allocated to another data processing system, and in a case where the at least one application is allocated to the other data processing system, (1) the application is started in the other data processing system, (2) context data about the application stored in a self system is transmitted to the other data processing system, and (3) processing of the application in the self system is stopped in response to a context switch at a certain timing of the application in the self system as a trigger.

17. A computer program for causing a data processing system including a GPU and an instruction section to execute the following steps, the GPU being a graphics processing unit: a step performed by the GPU to perform generation of processing of a plurality of pieces of data corresponding to a plurality of applications executed in parallel in a time-division manner; and a step performed by the instruction section to instruct the GPU to switch to generation of processing of an image corresponding to a second application different from a first application in a case where the GPU completes generation of processing of one data unit corresponding to the first application and to be handed over to a subsequent processing section; wherein wherein, in a case where a GPU occupancy time period of a predetermined number or more of the plurality of applications exceeds a predetermined allocation time period, at least one application of the plurality of applications is allocated to another data processing system, and in a case where the at least one application is allocated to the other data processing system, (1) the application is started in the other data processing system, (2) context data about the application stored in a self system is transmitted to the other data processing system, and (3) processing of the application in the self system is stopped in response to a context switch at a certain timing of the application in the self system as a trigger.

Images