Apparatus and method for rendering scheduling, and display system

Abstract

The invention discloses a rendering scheduling device and method and a display system. An apparatus for rendering scheduling, comprising: a data queue for holding a plurality of data structure instances, the data structure instances comprising information to be rendered to be rendered by one of a plurality of graphics processing units (GPUs) and timing information associated with a display timing of the information to be rendered; the scheduling unit is used for scheduling the plurality of GPUs to render the to-be-rendered information in the plurality of data structure instances so as to generate a plurality of image frames, and the to-be-rendered information in at least one part of the plurality of data structure instances is asynchronously rendered by the plurality of GPUs; and a synchronization unit for writing the plurality of image frames rendered by the plurality of GPUs back to the display device in an order indicated by the timing information for display on the display device.

Description

Technical Field

[0001] The present disclosure relates to the field of graphics rendering technology, and more specifically, to a device and method for rendering scheduling and a display system. Background Art

[0002] With the industrial upgrading of games, movies, 3D modeling, interior design and other fields, graphics rendering technology has received more and more attention and research. High quality and low latency are two important indicators in graphics rendering technology. When existing rendering systems process complex graphics and scenes, if they want to provide high-quality images, they often experience significant delays. This delay not only affects the user experience, but may also cause users to experience discomfort such as motion sickness. On the other hand, in order to reduce latency, existing systems may sacrifice image quality, resulting in blurred image details and inaccurate color reproduction in the final output, affecting the user's immersive experience.

[0003] At present, when rendering a large scene, such as a city block scene, or very fine object modeling, the delay generated by the rendering system becomes unacceptable. The increase in delay will cause freezes, ghosting, and other phenomena, which significantly affect the user experience. In order to reduce rendering delay, existing devices are studying parallel rendering through multiple graphics processing units (GPUs). In existing rendering schemes, bus communication between multiple GPUs can be used for synchronous parallel rendering, for example, CrossFire technology, SLI technology, etc. VR-NeRF is a technology that can use multiple GPUs for parallel rendering, which has achieved 36FPS in indoor scenes reconstructed and modeled by NeRF 3D. For virtual reality (VR) devices, a dual-GPU collaborative rendering method and device based on the UE engine DX12MutiAdapter (CN116883631A) uses two GPUs for parallel rendering, and the two GPUs render the left and right eye perspective images of the VR device respectively. The VR multi-graphics card rendering method and system (CN117094877A) divides the single-frame image rendering task and distributes it to multiple GPUs, using the parallel computing power of multiple GPUs to improve the binocular stereo rendering speed of the system.

[0004] However, existing solutions have strict requirements for achieving parallel rendering of multiple GPUs. For example, multiple GPUs must be of the same model, and the number of GPUs that the system can use is fixed. In addition, parallel rendering of multiple GPUs can only be synchronous rendering, resulting in a waste of computing power and large delays. Furthermore, splitting the single-frame image rendering task results in the rendering task being too fragmented after cutting, which is not convenient for overall optimization. Summary of the invention

[0005] It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed.

[0006] According to a first aspect of the present disclosure, a device for rendering scheduling is provided, comprising: a data queue, the data queue being used to maintain multiple data structure instances, the data structure instances including information to be rendered to be rendered by one of multiple graphics processing units (GPUs) and timing information associated with the display timing of the information to be rendered; a scheduling unit, the scheduling unit being used to schedule the multiple GPUs to render the information to be rendered in the multiple data structure instances to generate multiple image frames, wherein the information to be rendered in at least a portion of the multiple data structure instances is rendered asynchronously by the multiple GPUs; and a synchronization unit, the synchronization unit being used to write the multiple image frames rendered by the multiple GPUs back to a display device in an order indicated by the timing information for display on the display device.

[0007] In one embodiment, the scheduling unit is used to: calculate a current frame rate of the display device; and schedule the multiple GPUs based on the current frame rate.

[0008] In one embodiment, the scheduling unit is used to selectively adjust the number of enabled GPUs based on a comparison of the current frame rate with a predetermined frame rate range.

[0009] In one embodiment, the scheduling unit is used to: increase the number of enabled GPUs in response to the current frame rate being lower than a lower limit of the predetermined frame rate range; and reduce the number of enabled GPUs in response to the current frame rate being higher than an upper limit of the predetermined frame rate range.

[0010] In one embodiment, the GPU that is reduced includes the GPU that can provide the lowest frame rate.

[0011] In one embodiment, the adjusted number of enabled GPUs is a fixed value, or the adjusted number of enabled GPUs is determined based on the difference between the current frame rate and the lower limit value of the predetermined frame rate range or the difference between the current frame rate and the upper limit value of the predetermined frame rate range.

[0012] In one embodiment, the scheduling unit includes a rendering state machine, and the rendering state machine is used to selectively adjust the number of enabled GPUs.

[0013] In one embodiment, the predetermined frame rate range is determined based on the size and / or complexity of the scene to be rendered.

[0014] In one embodiment, the data queue is implemented through a shared memory, a file system, or a remote dictionary service cache.

[0015] In one embodiment, the synchronization unit is used to: obtain the timing information of the last image frame displayed by the display device from the scheduling unit; obtain the rendered data structure instance from the multiple GPUs; compare the timing information of the last image frame displayed with the timing information in the rendered data structure instance; and process the rendered data structure instance based on the result of the comparison.

[0016] In one embodiment, the synchronization unit is used to: in response to the time indicated by the timing information of the last displayed image frame being earlier than the time indicated by the timing information in the rendered data structure instance, write the image frame in the rendered data structure instance back to the display device; in response to the time indicated by the timing information of the last displayed image frame being later than the time indicated by the timing information in the rendered data structure instance, discard the rendered data structure instance.

[0017] In one embodiment, the timing information includes a timestamp.

[0018] In one embodiment, it further includes an acquisition unit, wherein the acquisition unit is used to: acquire the information to be rendered; encapsulate the information to be rendered and the timing information in the data structure instance; and store the data structure instance in the data queue.

[0019] According to a second aspect of the present disclosure, a display system is provided, comprising: a device for rendering scheduling as described in any of the foregoing items; and a plurality of graphics processing units (GPUs) coupled to the device, the plurality of GPUs being scheduled by the device to render data structure instances.

[0020] In one embodiment, it further includes: a display device coupled to the apparatus, wherein the display device is used to display the written-back image frame.

[0021] In one embodiment, the display device includes a virtual reality (VR) display device, and the VR display device includes a left-eye display device and a right-eye display device, wherein the to-be-rendered information associated with at least a portion of the image frames to be displayed on the left-eye display device and the to-be-rendered information associated with at least a portion of the image frames to be displayed on the right-eye display device are rendered asynchronously.

[0022] According to a third aspect of the present disclosure, a method for rendering scheduling is provided, comprising: maintaining a plurality of data structure instances by a data queue, the data structure instances comprising information to be rendered to be rendered by one of a plurality of graphics processing units (GPUs) and timing information associated with a display timing of the information to be rendered; scheduling the plurality of GPUs by a scheduling unit to render the information to be rendered in the plurality of data structure instances to generate a plurality of image frames, wherein the information to be rendered in at least a portion of the plurality of data structure instances is rendered asynchronously by the plurality of GPUs; and writing the plurality of image frames rendered by the plurality of GPUs back to a display device by a synchronization unit in an order indicated by the timing information for display on the display device.

[0023] In one embodiment, scheduling the multiple GPUs by the scheduling unit includes: calculating a current frame rate of the display device; and scheduling the multiple GPUs based on the current frame rate.

[0024] In one embodiment, scheduling the plurality of GPUs based on the current frame rate includes selectively adjusting a number of enabled GPUs based on a comparison of the current frame rate with a predetermined frame rate range.

[0025] In one embodiment, based on the comparison between the current frame rate and the predetermined frame rate range, selectively adjusting the number of enabled GPUs includes: in response to the current frame rate being lower than a lower limit of the predetermined frame rate range, increasing the number of enabled GPUs; in response to the current frame rate being higher than an upper limit of the predetermined frame rate range, reducing the number of enabled GPUs.

[0026] In one embodiment, the adjusted number of enabled GPUs is a fixed value, or the adjusted number of enabled GPUs is determined based on the difference between the current frame rate and the lower limit value of the predetermined frame rate range or the difference between the current frame rate and the upper limit value of the predetermined frame rate range.

[0027] In one embodiment, the synchronization unit writes the multiple image frames back to the display device in the order indicated by the timing information, including: obtaining the timing information of the last image frame displayed by the display device from the scheduling unit; obtaining the rendered data structure instance from the multiple GPUs; comparing the timing information of the last image frame displayed with the timing information in the rendered data structure instance; and processing the rendered data structure instance based on the result of the comparison.

[0028] In one embodiment, processing the rendered data structure instance based on the result of the comparison includes: in response to the time indicated by the timing information of the last image frame displayed being earlier than the time indicated by the timing information in the rendered data structure instance, writing the image frame in the rendered data structure instance back to the display device; in response to the time indicated by the timing information of the last image frame displayed being later than the time indicated by the timing information in the rendered data structure instance, discarding the rendered data structure instance.

[0029] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements any of the aforementioned methods when executed by a processor.

[0030] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the aforementioned methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are included to provide a further understanding of the present invention, and are incorporated into and constitute a part of this application. The accompanying drawings illustrate embodiments of the present invention and together with the specification serve to explain the principles of the present invention. In the accompanying drawings:

[0032] Figure 1 is a block diagram of a rendering scheduling device according to an embodiment of the present disclosure;

[0033] Figure 2 is a schematic diagram of a rendering state machine according to an embodiment of the present disclosure;

[0034] Figure 3 is a schematic diagram of writing an image frame in a rendered data structure instance back to a display device according to an embodiment of the present disclosure;

[0035] Figure 4 is a block diagram of a display system according to an embodiment of the present disclosure;

[0036] Figure 5 is a schematic diagram of a rendering architecture according to an embodiment of the present disclosure;

[0037] Figure 6 is a flowchart of a rendering scheduling method according to an embodiment of the present disclosure;

[0038] Figure 7 is a flowchart of a method for scheduling multiple GPUs by a scheduling unit according to an embodiment of the present disclosure;

[0039] Figure 8is a flowchart of a method for selectively adjusting the number of enabled GPUs according to one embodiment of the present disclosure; and

[0040] Fig. 9 The present invention is a flowchart of a method for writing, by a synchronization unit, a plurality of image frames back to a display device in a sequence indicated by timing information according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] Embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but the present invention is not limited thereto but only by the claims. In the accompanying drawings, for illustrative purposes, the sizes of some of the elements may be exaggerated and not drawn to scale. Wherever possible, the same reference numerals will be used in all drawings to represent the same or similar parts.

[0042] Although the terms used in the present invention are selected from well-known and commonly used terms, some of the terms mentioned in the present invention specification may be selected by the applicant at his or her discretion, and their detailed meanings are explained in the relevant parts of the description of this article. In addition, it is required to understand the present invention not only by the actual terms used, but also by the meaning contained in each term.

[0043] In the description provided herein, numerous specific details are set forth. However, it should be understood that embodiments of the present invention may be practiced without these specific details. In other examples, well-known methods, structures, and techniques are not shown in detail to avoid obscuring an understanding of the present invention.

[0044] According to one aspect of the present disclosure, a device for rendering scheduling is provided.

[0045] Figure 1 1 is a block diagram of a rendering scheduling device 100 according to an embodiment of the present disclosure. The rendering scheduling device 100 includes a data queue 102 , a scheduling unit 104 and a synchronization unit 106 .

[0046] The data queue 102 may maintain multiple data structure instances. The data structure instance may include information to be rendered by one of the multiple GPUs coupled to the rendering scheduling apparatus 100. For example, the information to be rendered may include display device information, viewing angle information, and the like.

[0047] The display device information may include the resolution and display refresh rate of the display device used to display the image frame. The display device may be, for example, a display of a computer, VR glasses, or other device, which may display the rendered image frame. When executing a rendering task, multiple GPUs coupled to the rendering scheduling device 100 may set specific parameters of the rendering task according to the display device information. For example, the higher the resolution and display refresh rate of the display device, the greater the amount of computation required for rendering.

[0048] The viewing angle information is used to describe the viewing angle of the subject for which the image frame displayed by the display device is directed. The viewing angle information may include the viewing angle position and the viewing angle direction. For example, when a user wears a VR device to play a game, the user's own viewing angle is consistent with the viewing angle of the device. For example, when the user's head turns, the viewing angle of the image displayed in the VR device changes with the rotation of the head. Therefore, the physical position of the VR device can be converted into a virtual position in the VR device's own coordinate system, that is, the viewing angle position, and the user's head orientation can be converted into the viewing angle orientation. For example, when a user uses an input device such as a keyboard or a joystick to play a game, the user's operation of the input device can change the position and orientation. Therefore, the viewing angle position and viewing angle orientation can be calculated based on the initial position and initial orientation, based on information such as the operation amplitude and number of operations of the input device.

[0049] The data structure instance may also include timing information associated with the display timing of the information to be rendered. In some embodiments, the timing information may include a timestamp. The timing information may correspond to the time when the user performs the action, or may correspond to the time when the viewing angle information is acquired, or may correspond to any suitable time sequence reflecting the display of the image frames on the display device.

[0050] In some embodiments, the data queue 102 can be implemented through shared memory, file system, remote dictionary service (Remote Dictionary Server, Redis) cache, etc.

[0051] As an example, when a user wears VR glasses to play games, when the user's head turns to the left, a motion track is generated, and the VR glasses will sample and render multiple points on the motion track for final display on the display of the VR glasses. By sampling multiple points in the motion track and obtaining the information to be rendered of each point, multiple data structure instances containing the information to be rendered and timing information can be generated, and these data structure instances can be stored in the data queue 102 for multiple GPUs to render.

[0052] The scheduling unit 104 can schedule multiple GPUs to render the information to be rendered in the multiple data structure instances in the data queue 102 to generate multiple image frames. The information to be rendered in at least a part of the multiple data structure instances is rendered asynchronously by multiple GPUs. Synchronous rendering means that only after the rendering task of the first image frame is completed, the rendering task of the second image frame can be executed. Different from synchronous rendering, asynchronous rendering means that the rendering task is independent of the first image frame, and the rendering task of the second image frame can be executed regardless of whether the rendering task of the first image frame has been completed. For the multiple data structure instances stored in the data queue 102, the scheduling unit 104 can schedule multiple GPUs in sequence, each GPU can take out a data structure instance from the data queue 102, and render the information to be rendered in the data structure instance to generate an image frame, thereby realizing asynchronous rendering of different data structure instances. The rendered image frame is stored in an image buffer. The data structure instance can store an image buffer pointer, which is used to point to the rendered image frame stored in the image buffer.

[0053] The synchronization unit 106 may write the image frames rendered by the multiple GPUs back to the display device in the order indicated by the timing information for display on the display device.

[0054] In some embodiments, the scheduling unit 104 and the synchronization unit 106 may be implemented by components such as a processor, a field-programmable gate array (FPGA), and an application-specific integrated circuit (ASIC).

[0055] In some embodiments, the rendering scheduling device 100 may further include an acquisition unit. The acquisition unit may acquire information to be rendered. For example, the acquisition unit may acquire display device information from a display device coupled to the rendering scheduling device 100, and may acquire viewing angle information from a detection device coupled to the rendering scheduling device 100. For example, when the rendering scheduling device 100 is used for VR glasses, display device information may be acquired from the display device of the VR glasses, and viewing angle information may be acquired from a detection device (e.g., a camera or a sensor) of the VR glasses. For example, when the rendering scheduling device 100 is used for a computer, display device information may be acquired from a display device (e.g., a display screen) of the computer, and viewing angle information may be acquired from a detection device (e.g., a sensor corresponding to an input device, such as a keyboard, a mouse, etc.) of the computer. Then, the acquisition unit may encapsulate the information to be rendered and the corresponding timing information in a data structure instance, and store the data structure instance in the data queue 102.

[0056] As an example, a packaged data structure is as follows:

[0057]

[0058] In this example, uint64_t timestamp may be used to store timing information (eg, a timestamp) or point to a location where a timestamp is stored. After rendering, the rendered image frame may be stored in the resource pointed to by the pointer image_buffer and accessed through image_cuda_handle.

[0059] In the existing scheme, the GPU is usually used to perform synchronous rendering. Only after the rendering task of the first image frame is completed, the rendering task of the second image frame can be continued, resulting in only one GPU being used to perform the rendering task at the same time, low GPU utilization, and thus high latency. The scheme disclosed in the present invention realizes asynchronous rendering by introducing a data queue to store data structure instances corresponding to different information to be rendered, so that when the first GPU performs the rendering task of the first image frame, the second GPU can simultaneously perform the rendering task of the second image frame, thereby making full use of the GPU computing power and greatly reducing display latency.

[0060] Since the frame rate of the display device is constantly changing, if the same number of GPUs is always enabled, when the frame rate exceeds the display requirement, that is, the number of GPUs executing the rendering task is too large, it will lead to a waste of GPU computing power, and when the frame rate does not meet the display requirement, that is, the number of GPUs executing the rendering task is too small, the GPU computing power is not fully utilized, which will lead to high latency, resulting in a poor user experience. In order to further achieve full utilization and conservation of the computing power of multiple GPUs, in some embodiments, the scheduling unit 104 can calculate the current frame rate of the display device coupled to the rendering scheduling device 100. The current frame rate can be calculated as the average frame rate of the display device in a time period, and the frame rate at each time point in the time period can be calculated based on the number of image frames displayed by the display device within a certain period of time, or can be calculated based on the number of image frames written back to the display device by the synchronization unit 106 within a certain period of time. Then, the scheduling unit 104 can schedule multiple GPUs coupled to the rendering scheduling device 100 based on the current frame rate of the display device.

[0061] In some embodiments, the scheduling unit 104 may selectively adjust the number of enabled GPUs based on a comparison of the current frame rate with a predetermined frame rate range. In some embodiments, the predetermined frame rate range may be determined based on the size of the scene to be rendered. For example, when the display device is a computer monitor, the standard frame rate of the monitor is 30FPS, then the predetermined frame rate range may be set to 24-48FPS; when the display device is a monitor of a VR device, the standard frame rate of the monitor is 72FPS, then the predetermined frame rate range may be set to 60-120FPS. In some embodiments, the predetermined frame rate range may be determined based on the complexity of the scene to be rendered. For example, for scenes such as large games and 3D modeling, the predetermined frame rate range may be set to a higher range, while for slowly changing scenes, the predetermined frame rate range may be set to a lower range. The predetermined frame rate range may be a default value, or it may be set by the user as needed.

[0062] In some embodiments, the scheduling unit 104 may increase the number of enabled GPUs in response to the current frame rate being lower than the lower limit of the predetermined frame rate range. When the number of enabled GPUs is less than the number of all GPUs, that is, when there are unenabled GPUs, more GPUs may be enabled to increase the frame rate.

[0063] In some embodiments, the scheduling unit 104 may reduce the number of enabled GPUs in response to the current frame rate being higher than the upper limit of the predetermined frame rate range. When the number of enabled GPUs is greater than one, at least one enabled GPU may be disabled to reduce the frame rate. After reducing the number of enabled GPUs, it is necessary to ensure that there is at least one enabled GPU to perform the rendering task.

[0064] In some embodiments, the GPU that is reduced may include the GPU that can provide the lowest frame rate. When the current frame rate is higher than the upper limit of the predetermined frame rate range, if some GPUs are to be disabled, the GPU that can provide the lowest frame rate, that is, the GPU with the lowest computing power, may be disabled first, thereby making better use of the GPU with higher computing power, so that the current frame rate can be more stably maintained within the predetermined frame rate range, thereby effectively avoiding the need to schedule GPUs back and forth.

[0065] In some embodiments, the scheduling unit 104 may maintain the number of enabled GPUs in response to the current frame rate being within a predetermined frame rate range.

[0066] In some embodiments, the adjusted number of enabled GPUs may be a fixed value. For example, if the current frame rate is lower than the lower limit of the predetermined frame rate range, a fixed number of GPUs may be enabled each time, such as one, two, etc. If the current frame rate is higher than the upper limit of the predetermined frame rate range, a fixed number of GPUs may be disabled each time, such as one, two, etc. Therefore, the number of enabled GPUs may be increased or decreased by a fixed value according to the relationship between the current frame rate and the predetermined frame rate, so that the adjusted frame rate is within the predetermined frame rate range.

[0067] In some embodiments, the number of GPUs enabled to be adjusted is determined based on the difference between the current frame rate and the lower limit of the predetermined frame rate range. For example, if the current frame rate is 20 FPS, the predetermined frame rate range is set to 60-80 FPS, and assuming that the frame rate that each GPU can provide is 30 FPS, the number of GPUs enabled to be increased is two.

[0068] In some embodiments, the number of GPUs enabled to be adjusted is determined based on the difference between the current frame rate and the upper limit of the predetermined frame rate range. For example, if the current frame rate is 90 FPS, the predetermined frame rate range is set to 60-80 FPS, and assuming that the frame rate that each GPU can provide is 30 FPS, the number of GPUs enabled to be reduced is one.

[0069] When multiple GPUs are capable of providing different frame rates, the scheduling unit can flexibly select to enable or disable a GPU as needed.

[0070] In some embodiments, the scheduling unit 104 may include a rendering state machine for selectively adjusting the number of enabled GPUs. Figure 2 1 is a schematic diagram of a rendering state machine according to an embodiment of the present disclosure. After the rendering state machine is started, if the current frame rate is lower than the lower limit of the predetermined frame rate range, the number of enabled GPUs is increased. If the current frame rate is within the predetermined frame rate range, the number of enabled GPUs is maintained. If the current frame rate is higher than the upper limit of the predetermined frame rate range, the number of enabled GPUs is reduced.

[0071] Therefore, the solution disclosed in the present invention can adjust the computing power according to the rendering requirements, increase the number of enabled GPUs when the current frame rate does not meet the rendering requirements, and reduce the number of enabled GPUs when the current frame rate exceeds the rendering requirements, so that smooth rendering can be achieved for large and complex scenes, and computing resources can be saved and costs can be reduced for small and simple scenes. Therefore, the solution disclosed in the present invention solves the problems of excessive waste and insufficient computing power by realizing flexible parallel rendering, and achieves smooth rendering under different computing powers.

[0072] On the basis of parallel rendering by multiple GPUs, since the computing performance of different GPUs is not exactly the same, for example, different models of GPUs have different computing power, and the same model of GPUs have different computing power due to different service times, etc., different GPUs may take different times to execute the rendering task of a single data structure instance. If the first GPU with lower computing power is executing the rendering of the first data structure instance at an earlier time, and the second GPU with higher computing power has completed the rendering of the second data structure instance at a later time, since the rendering of the first data structure instance has not yet been completed, it is necessary to wait for the rendering of the first data structure instance at an earlier time to be completed and displayed before the image frame in the second data structure instance rendered at a later time can be displayed, resulting in a delay in image display and a poor user experience. In order to reduce delays, the present disclosure utilizes a synchronization unit to execute an innovative synchronization strategy to write the image frames in the rendered data structure instance back to the display device in a timely manner.

[0073] In some embodiments, the synchronization unit 106 may obtain the timing information of the last image frame displayed by the display device from the scheduling unit 104. Then, the rendered data structure instance is obtained from the multiple GPUs. Next, the timing information of the last image frame displayed is compared with the timing information in the rendered data structure instance. Finally, the rendered data structure instance is processed based on the comparison result.

[0074] In some embodiments, the synchronization unit 106 may write the image frame in the rendered data structure instance back to the display device in response to the time indicated by the timing information of the last displayed image frame being earlier than the time indicated by the timing information in the rendered data structure instance.

[0075] In some embodiments, the synchronization unit 106 may discard the rendered data structure instance in response to the time indicated by the timing information of the last displayed image frame being later than the time indicated by the timing information in the rendered data structure instance.

[0076] Figure 31 is a schematic diagram of writing an image frame in a rendered data structure instance back to a display device according to an embodiment of the present disclosure. For example, the synchronization unit 106 may obtain the timing information of the last image frame displayed by the display device from the scheduling unit 104, which is indicated by a slash on the left side of part (a), and the timing information is #100. If the second GPU with higher computing power completes the rendering of the second data structure instance at a later time first, as indicated by a square on the right side of part (a), and the timing information is #104, and the first GPU with lower computing power completes the rendering of the first data structure instance at an earlier time later, as indicated by dots on the right side of part (a), and the timing information is #102, then the synchronization unit 106 may first obtain the rendered second data structure instance from the second GPU, as indicated by a square on the right side of part (a), and the timing information is #104. Since the timing information #104 in the second data structure instance is later than the timing information #100 of the last image frame that has been displayed, the synchronization unit 106 can write the rendered image frame in the second data structure instance back to the display device for display on the display device. Then, the scheduling unit 104 can update the timing information of the last image frame that has been displayed to the timing information #104 in the second data structure instance that has been displayed, as shown on the left side of part (b). After the second GPU completes rendering of the second data structure instance, the second GPU can continue to execute the next rendering task.

[0077] Next, the synchronization unit 106 can obtain the timing information of the last image frame displayed by the display device from the scheduling unit 104, that is, the timing information #104 in the second data structure instance that has been updated and displayed previously. Then, the synchronization unit 106 can obtain the rendered first data structure instance from the first GPU, as indicated by dots on the right side of part (b), and the timing information is #102. However, since the timing information #102 in the first data structure instance is earlier than the timing information #104 in the second data structure instance that has been displayed, that is, compared with the image frame in the newly acquired first data structure instance, the display device has displayed the latest image frame. At this time, if the image frame in the first data structure instance with an earlier time is displayed, it will cause time reversal in the display, and the user experience is not good. Therefore, the synchronization unit 106 can discard the first data structure instance, and the timing information of the last image frame that has been displayed remains #104, as shown on the left side of part (c). Subsequently, the rendered data structure instance is processed in a similar manner.

[0078] Compared with waiting for the rendering of each data structure instance to be completed in the order indicated by the timing information, the above synchronization strategy can reduce the delay between user operation and image frame display, further improving the user experience.

[0079] According to another aspect of the present disclosure, a display system is provided.

[0080] Figure 4 4 is a block diagram of a display system 400 according to an embodiment of the present disclosure. The display system 400 includes a rendering scheduling device 402 and a plurality of GPUs 404-1 to 404-N. The rendering scheduling device 402 may be based on Figure 1 The rendering scheduling device 100 described above. A plurality of GPUs 404-1 to 404-N are coupled to the rendering scheduling device 402 and are scheduled by the rendering scheduling device 402 to render data structure instances in a data queue in the rendering scheduling device 402. In some embodiments, at least two GPUs among the plurality of GPUs 404-1 to 404-N may have the same or different computing capabilities.

[0081] In some embodiments, the display system 400 may further include a display device 406. The display device 406 is coupled to the rendering scheduling apparatus 402 and is used to display the written-back image frame. The display device 406 may be, for example, a display device of a computer, a VR device, or the like.

[0082] In some embodiments, the display device 406 may include a VR display device. In some embodiments, the VR display device may include a left-eye display device and a right-eye display device. Wherein, the information to be rendered associated with at least a portion of the image frames to be displayed on the left-eye display device and the information to be rendered associated with at least a portion of the image frames to be displayed on the right-eye display device are rendered asynchronously. In some embodiments, the information to be rendered associated with at least a portion of the image frames to be displayed on the left-eye display device and the information to be rendered associated with at least a portion of the image frames to be displayed on the right-eye display device are rendered independently of each other.

[0083] Figure 5 is a schematic diagram of a rendering architecture according to an embodiment of the present disclosure. An acquisition unit acquires information to be rendered from a display device, encapsulates the information to be rendered and corresponding timing information in a data structure instance, and stores the data structure instance in a data queue. The data queue maintains multiple data structure instances for rendering by multiple GPUs. A scheduling unit schedules multiple GPUs to render the information to be rendered in multiple data structure instances to generate multiple image frames. A synchronization unit writes multiple image frames rendered by multiple GPUs back to the display device in an order indicated by the timing information for display on the display device.

[0084] According to another aspect of the present disclosure, a method for rendering scheduling is provided.

[0085] Figure 66 is a flowchart of a rendering scheduling method 600 according to an embodiment of the present disclosure. The rendering scheduling method 600 may be implemented by components such as a processor, a field programmable gate array (FPGA), and an application specific integrated circuit (ASIC).

[0086] At step 602, a data queue may maintain multiple data structure instances. The data structure instances include information to be rendered by one of a plurality of graphics processing units (GPUs) and timing information associated with the display timing of the information to be rendered. In some embodiments, the timing information includes a timestamp.

[0087] At step 604, the scheduling unit may schedule multiple GPUs to render the information to be rendered in the multiple data structure instances to generate multiple image frames. The information to be rendered in at least a portion of the multiple data structure instances is rendered asynchronously by the multiple GPUs.

[0088] At step 606 , the synchronization unit may write the multiple image frames rendered by the multiple GPUs back to the display device in the order indicated by the timing information for display on the display device.

[0089] By using data queues to store data structure instances corresponding to different information to be rendered, asynchronous rendering can be achieved, which can fully utilize the GPU computing power, reduce display delays, and optimize user experience.

[0090] Figure 7 is a flowchart of a method 700 for scheduling multiple GPUs by a scheduling unit according to an embodiment of the present disclosure.

[0091] At step 702, the current frame rate of the display device may be calculated.

[0092] At step 704, multiple GPUs may be scheduled based on the current frame rate.

[0093] In some embodiments, step 704 may include step 710. At step 710, the number of enabled GPUs may be selectively adjusted based on a comparison of the current frame rate with a predetermined frame rate range. In some embodiments, the predetermined frame rate range is determined based on the size and / or complexity of the scene to be rendered.

[0094] By scheduling multiple GPUs based on the current frame rate of the display device, flexible rendering can be achieved, thereby making full use of and saving computing power according to different computing power requirements.

[0095] Figure 8is a flow chart of a method 800 of selectively adjusting the number of enabled GPUs according to one embodiment of the present disclosure.

[0096] At step 802, a current frame rate may be compared to a predetermined frame rate range.

[0097] In response to the current frame rate being lower than a lower limit of a predetermined frame rate range at step 802 , at step 804 , the number of enabled GPUs may be increased.

[0098] In response to the current frame rate being higher than an upper limit of the predetermined frame rate range at step 802 , at step 806 , the number of enabled GPUs may be reduced.

[0099] In response to the current frame rate being within the predetermined frame rate range at step 802 , at step 808 , the number of enabled GPUs may be maintained.

[0100] In some embodiments, the GPU that is reduced includes the GPU that is capable of providing the lowest frame rate.

[0101] In some embodiments, the adjusted number of enabled GPUs is a fixed value. In some embodiments, the adjusted number of enabled GPUs can be determined based on the difference between the current frame rate and the lower limit of the predetermined frame rate range or the difference between the current frame rate and the upper limit of the predetermined frame rate range.

[0102] In some embodiments, the scheduling unit may include a rendering state machine for selectively adjusting the number of enabled GPUs.

[0103] Fig. 9 is a flowchart of a method 900 for writing, by a synchronization unit, a plurality of image frames back to a display device in an order indicated by timing information according to an embodiment of the present disclosure.

[0104] At step 902, timing information of the last image frame displayed by the display device may be obtained from the scheduling unit.

[0105] At step 904 , rendered data structure instances may be obtained from a plurality of GPUs.

[0106] At step 906, the timing information of the last image frame that was displayed may be compared with the timing information in the rendered data structure instance.

[0107] At step 908 , the rendered data structure instance may be processed based on the results of the comparison.

[0108] In some embodiments, processing the rendered data structure instance based on the comparison result includes: in response to the time indicated by the timing information of the last image frame displayed being earlier than the time indicated by the timing information in the rendered data structure instance, writing the image frame in the rendered data structure instance back to the display device.

[0109] In some embodiments, processing the rendered data structure instance based on the comparison result includes: discarding the rendered data structure instance in response to the time indicated by the timing information of the last image frame displayed being later than the time indicated by the timing information in the rendered data structure instance.

[0110] By writing back the rendered image frames based on timing information, the delay between user operations and image frame display can be reduced, thereby improving the user experience.

[0111] In addition, the present disclosure also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the above method when executed by a processor.

[0112] In addition, the present disclosure also provides a computer program product, including a computer program, wherein the computer program implements the above method when executed by a processor.

[0113] It has been experimentally verified that the implementation scheme of the present disclosure can perform smooth rendering on city-level scenes, and computing resources can be automatically adjusted according to the rendering computing power requirements.

[0114] In summary, the embodiments of the present disclosure provide an asynchronous rendering technology that supports rendering in scenes of various scales. The present disclosure introduces a data queue to store data structure instances, so that multiple GPUs can perform asynchronous rendering on different data structure instances, thereby reducing delays and providing a smooth user experience. The present disclosure schedules multiple GPUs based on the current frame rate, so that the number of enabled GPUs can be flexibly adjusted, thereby achieving elastic scheduling of computing resources while ensuring that the frame rate meets the display requirements. When the frame rate does not meet the display requirements, by scheduling more computing resources for rendering, it is possible to reduce delays, make full use of computing resources, and improve user experience. When the frame rate far exceeds the display requirements, by recycling some computing resources, it is possible to save overhead while meeting the display requirements. Since the embodiments of the present disclosure can realize asynchronous rendering and / or elastic rendering, resources can be dynamically scheduled according to the calculation amount of different scenes, and the frame rate flow of real-time rendering can be achieved, so it is suitable for complex scenes at the city block level, real-time rendering scenes, etc.

[0115] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0116] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0117] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0118] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0119] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0120] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0121] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0122] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0123] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A device for rendering scheduling, comprising: A data queue, the data queue being used to hold a plurality of data structure instances, the data structure instances comprising information to be rendered by one of a plurality of graphics processing units (GPUs) and timing information associated with a display timing of the information to be rendered; a scheduling unit, the scheduling unit being used to schedule the multiple GPUs to render the information to be rendered in the multiple data structure instances to generate multiple image frames, wherein the information to be rendered in at least a portion of the multiple data structure instances is rendered asynchronously by the multiple GPUs; as well as A synchronization unit is configured to write the plurality of image frames rendered by the plurality of GPUs back to a display device in an order indicated by the timing information, so as to be displayed on the display device.

2. The device according to claim 1, wherein: The scheduling unit is used for: Calculating a current frame rate of the display device; and The plurality of GPUs are scheduled based on the current frame rate.

3. The device as claimed in claim 2, wherein: The scheduling unit is used for: Based on a comparison of the current frame rate with a predetermined frame rate range, the number of enabled GPUs is selectively adjusted.

4. The device according to claim 3, wherein: The scheduling unit is used for: In response to the current frame rate being lower than a lower limit of the predetermined frame rate range, increasing the number of enabled GPUs; In response to the current frame rate being higher than an upper limit of the predetermined frame rate range, the number of enabled GPUs is reduced.

5. The device according to claim 4, wherein: The GPUs that are reduced include the GPU that can provide the lowest frame rate.

6. The device according to claim 3, wherein: The number of GPUs enabled is adjusted to a fixed value, or The adjusted number of enabled GPUs is determined based on a difference between the current frame rate and a lower limit value of the predetermined frame rate range or a difference between the current frame rate and an upper limit value of the predetermined frame rate range.

7. The device of claim 3, wherein: The scheduling unit includes a rendering state machine for selectively adjusting the number of enabled GPUs.

8. The device of claim 3, wherein: The predetermined frame rate range is determined based on the size and / or complexity of the scene to be rendered.

9. The device of claim 1, wherein: The data queue is implemented through a shared memory, a file system, or a remote dictionary service cache.

10. The device of claim 1, wherein: The synchronization unit is used for: Acquire timing information of a previous image frame displayed by the display device from the scheduling unit; Obtain rendered data structure instances from the plurality of GPUs; Comparing the timing information of the last displayed image frame with the timing information in the rendered data structure instance; as well as The rendered data structure instance is processed based on a result of the comparison.

11. The device of claim 10, wherein: The synchronization unit is used for: In response to the time indicated by the timing information of the last displayed image frame being earlier than the time indicated by the timing information in the rendered data structure instance, writing the image frame in the rendered data structure instance back to the display device; In response to the time indicated by the timing information of the last displayed image frame being later than the time indicated by the timing information in the rendered data structure instance, the rendered data structure instance is discarded.

12. The device of claim 1, wherein: The timing information includes a timestamp.

13. The device of claim 1, wherein: The invention also includes an acquisition unit, wherein the acquisition unit is used to: Obtaining the information to be rendered; Encapsulating the information to be rendered and the timing information in the data structure instance; as well as The data structure instance is stored in the data queue.

14. A display system comprising: The device for rendering scheduling according to any one of claims 1 to 13; as well as A plurality of graphics processing units (GPUs) are coupled to the device, and the plurality of GPUs are scheduled by the device to render the data structure instance.

15. The display system according to claim 14, wherein: Also includes: A display device is coupled to the apparatus, and is used for displaying the written-back image frame.

16. The display system according to claim 15, wherein: The display device includes a virtual reality (VR) display device, and the VR display device includes a left-eye display device and a right-eye display device. The to-be-rendered information associated with at least a portion of the image frames to be displayed on the left-eye display device and the to-be-rendered information associated with at least a portion of the image frames to be displayed on the right-eye display device are rendered asynchronously.

17. A method for rendering scheduling, comprising: A data queue maintains a plurality of data structure instances, wherein the data structure instances include information to be rendered by one of a plurality of graphics processing units (GPUs) and timing information associated with a display timing of the information to be rendered; The scheduling unit schedules the multiple GPUs to render the information to be rendered in the multiple data structure instances to generate multiple image frames, wherein the information to be rendered in at least a portion of the multiple data structure instances is rendered asynchronously by the multiple GPUs; as well as The plurality of image frames rendered by the plurality of GPUs are written back to a display device by a synchronization unit in an order indicated by the timing information for display on the display device.

18. The method of claim 17, wherein: Scheduling the multiple GPUs by the scheduling unit includes: Calculating a current frame rate of the display device; and The plurality of GPUs are scheduled based on the current frame rate.

19. The method of claim 18, wherein: Scheduling the plurality of GPUs based on the current frame rate includes: Based on a comparison of the current frame rate with a predetermined frame rate range, the number of enabled GPUs is selectively adjusted.

20. The method of claim 19, wherein: Based on the comparison of the current frame rate with the predetermined frame rate range, selectively adjusting the number of enabled GPUs includes: In response to the current frame rate being lower than a lower limit of the predetermined frame rate range, increasing the number of enabled GPUs; In response to the current frame rate being higher than an upper limit of the predetermined frame rate range, the number of enabled GPUs is reduced.

21. The method of claim 19, wherein: The number of GPUs enabled is adjusted to a fixed value, or The adjusted number of enabled GPUs is determined based on a difference between the current frame rate and a lower limit value of the predetermined frame rate range or a difference between the current frame rate and an upper limit value of the predetermined frame rate range.

22. The method of claim 17, wherein: Writing the plurality of image frames back to the display device in the order indicated by the timing information by the synchronization unit comprises: Acquire timing information of a previous image frame displayed by the display device from the scheduling unit; Obtain rendered data structure instances from the plurality of GPUs; Comparing the timing information of the last displayed image frame with the timing information in the rendered data structure instance; and The rendered data structure instance is processed based on a result of the comparison.

23. The method of claim 22, wherein: Processing the rendered data structure instance based on the result of the comparison includes: In response to the time indicated by the timing information of the last displayed image frame being earlier than the time indicated by the timing information in the rendered data structure instance, writing the image frame in the rendered data structure instance back to the display device; In response to the time indicated by the timing information of the last displayed image frame being later than the time indicated by the timing information in the rendered data structure instance, the rendered data structure instance is discarded.

24. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 17 to 23 is implemented.

25. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 17 to 23 is implemented.

Images