Industrial visualization using graphics processing hardware

Abstract

Methods for real-time simulation of a physical environment and corresponding systems, devices, and computer-readable mediums. A method performed by a graphical processing unit (GPU) (200) of a computer system (100) includes reading (402) simulation parameters from a uniform buffer (216) for processing by a compute pipeline (302) of a GPU (200) and by a graphics pipeline (304) of the GPU (200). The method includes programming (404) a vertex buffer (220) in the graphics pipeline (304), based on the simulation parameters, to produce first vertex buffer contents. The method includes performing a simulation (406) in the compute pipeline (302), based on the parameters, the first vertex buffer contents, and first storage buffer contents to produce second storage buffer contents, storing (408) the second storage buffer content in a storage buffer (222), and writing (410) second vertex buffer contents to the vertex buffer (220) based on the second storage buffer contents. The method includes writing (414) data to a frame buffer (210) for display to a user, in the graphics pipeline (304), based on the second vertex buffer contents.

Description

INDUSTRIAL VISUALIZATION USING GRAPHICS PROCESSING HARDWARETECHNICAL FIELD

[0001] The present disclosure is directed, in general, to systems and methods for visualization and simulation of industrial systems.BACKGROUND OF THE DISCLOSURE

[0002] The notion of “Industrial Metaverse” represents an assembly of technologies to enable real-time design, simulation and visualization in a common virtual place that can be reached simultaneously by multiple collaborators, in a way that is easily comprehensible by non-experts in the simulation technology. Current systems, however, are not easily implemented on readily-available hardware systems, and so designing, visualizing, or simulating industrial systems is inaccessible to the average user. Improved systems are desirable.SUMMARY OF THE DISCLOSURE

[0003] Various disclosed embodiments include methods for real-time simulation of a physical environment and corresponding systems, devices, and computer-readable mediums. A method performed by a graphical processing unit (GPU) of a computer system includes reading simulation parameters from a uniform buffer for processing by a compute pipeline of the GPU and by a graphics pipeline of the GPU. The method includes programming a vertex buffer in the graphics pipeline, based on the simulation parameters, to produce first vertex buffer contents. The method includes performing a simulation in the compute pipeline, based on the parameters, the first vertex buffer contents, and first storage buffer contents to produce second storage buffer contents, storing the second storage buffer content in a storage buffer, and writing second vertex buffer contents to the vertex buffer based on the second storage buffer contents. The method includes writing data to a frame buffer for display to a user, in the graphics pipeline, based on the second vertex buffer contents.

[0004] Various embodiments also include performing post processing, in the compute pipeline, based on the second vertex buffer contents to produce texture data, and wherein writing data to the frame buffer is also based on the texture data. In various embodiments, the simulation is a physics-based simulation of a physical environment and can be a physics-based simulation of an industrial operation.

[0005] In various embodiments, the GPU executes the compute pipeline and the graphics pipeline independently of each other with respect to time steps. In various embodiments, performing the simulation uses the first vertex buffer contents as geometry inputs. In various embodiments, the simulation is performed by a compute shader unit.

[0006] Disclosed embodiments include a computer system having a processor, an accessible memory, and a GPU, where the computer system and / or the GPU is configured to perform processes as described herein.

[0007] Disclosed embodiments include a GPU that includes a GPU processing unit having at least one GPU processor with multiple cores, configured to perform processes as described herein. Disclosed embodiments also include anon-transitory computer-readable medium encoded with executable instructions that, when executed, cause one or more GPUs to perform processes as described herein.

[0008] The foregoing has outlined rather broadly the features and technical advantages of the present disclosure so that those skilled in the art may better understand the detailed description that follows. Additional features and advantages of the disclosure will be described hereinafter that form the subject of the claims. Those skilled in the art will appreciate that they may readily use the conception and the specific embodiment disclosed as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Those skilled in the ail will also realize that such equivalent constructions do not depart from the spirit and scope of the disclosure in its broadest form.

[0009] Before undertaking the DETAILED DESCRIPTION below, it may be advantageous to set forth definitions of certain words or phrases used throughout this patent document: the terms “include” and “comprise,” as well as derivatives thereof, mean inclusion without limitation; the term “or” is inclusive, meaning and / or; the phrases “associated with” and “associated therewith,” as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like; and the term “controller” means any device, system or part thereof that controls at least one operation, whether such a device is implemented in hardware, firmware, software or some combination of at least two of the same. It should be noted that the functionality associated with any particular controller may be centralized or distributed, whether locally or remotely. Definitions for certain words and phrases are provided throughout this patent document, and those of ordinary skill in the art will understand that such definitions apply in many, if not most, instances to prior as well as future uses of such defined words andphrases. While some terms may include a wide variety of embodiments, the appended claims may expressly limit these terms to specific embodiments.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] For a more complete understanding of the present disclosure, and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, wherein like numbers designate like objects, and in which:

[0011] FIG. 1 illustrates a block diagram of a computer system in which an embodiment can be implemented;

[0012] FIG. 2 illustrates a block diagram of elements of a GPU in accordance with disclosed embodiments;

[0013] FIG. 3 illustrates an architecture and process for real-time design, visualization, and simulation of a physical environment, such as an industrial system, in accordance with disclosed embodiments; and

[0014] FIG. 4 illustrates a flowchart of a process in accordance with disclosed embodiments.DETAILED DESCRIPTION

[0015] FIGS. 1 through 4, discussed below, and the various embodiments used to describe the principles of the present disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably arranged device. The numerous innovative teachings of the present application will be described with reference to exemplary non-limiting embodiments.

[0016] Visualization of 3D virtual scenes and geometrical design are in the domain of computer graphics. In contrast, physics-based simulation is in the domain of computational sciences. The use of a Graphical Processing Unit (GPU) to accelerate the computation of a simulation is disconnected from the domain of graphics. In principle, the GPU processes the data in the memory buffer either through a graphics pipeline, for example to build a visual representation from geometrical data, or either through a compute pipeline, where the data are only accessed to perform mathematical operations.

[0017] Systems that simulate physical environments and systems are both graphically and computationally intensive. A solution to perform computation and visualization at the same time and interact in real-time with the data in memory has not been developed in the industry, since the graphical / display processing of such current systems are logically and often physically separated from each other.

[0018] Disclosed embodiments enable real-time interaction of a user with a simulation of a physics-based problem, visualization of post-processed simulation results, and real-time design parameter editing. Disclosed embodiments also use digital twins as a service in a web-based ecosystem for real-time design, visualization and simulation in an industrial metaverse.

[0019] FIG. 1 illustrates a block diagram of a computer system in which an embodiment can be implemented, for example as a computer system particularly configured by software or otherwise to perform the processes as described herein, and in particular as each one ofa plurality of interconnected and communicating systems as described herein. The computer system depicted includes a processor 102 connected to a level two cache / bridge 104, which is connected in turn to a local system bus 106. Local system bus 106 may be, for example, a peripheral component interconnect (PCI) architecture bus. Also connected to local system bus in the depicted example are a main memory 108 and a graphics adapter 110. The graphics adapter 110 may be connected to display 111. In particular, graphics adapter 110 can include specific elements, processors, buffers, memories, and other elements as described herein.

[0020] Other peripherals, such as local area network (LAN) / Wide Area Network I Wireless (e.g. WiFi) adapter 112, may also be connected to local system bus 106. Expansion bus interface 114 connects local system bus 106 to input / output (I / O) bus 116. I / O bus 116 is connected to keyboard / mouse adapter 118, disk controller 120, and I / O adapter 122. Disk controller 120 can be connected to a storage 126, which can be any suitable machine usable or machine readable storage medium, including but not limited to nonvolatile, non-transitory, hard-coded type mediums such as read only memories (ROMs) or erasable, electrically programmable read only memories (EEPROMs), magnetic tape storage, and user-recordable type mediums such as floppy disks, hard disk drives and compact disk read only memories (CD-ROMs) or digital versatile disks (DVDs), and other known optical, electrical, or magnetic storage devices.

[0021] Also connected to I / O bus 116 in the example shown is audio adapter 124, to which speakers (not shown) may be connected for playing sounds. Keyboard / mouse adapter 118 provides a connection for a pointing device (not shown), such as a mouse, trackball, trackpointer, touchscreen, etc.

[0022] Those of ordinary skill in the ail will appreciate that the hardware depicted in FIG. 1 may vary for particular implementations. For example, other peripheral devices, such as an optical disk drive and the like, also may be used in addition or in place of the hardware depicted. The depicted example is provided for the purpose of explanation only and is not meant to imply architectural limitations with respect to the present disclosure.

[0023] A computer system in accordance with an embodiment of the present disclosure includes an operating system employing a graphical user interface. The operating system permits multiple display windows to be presented in the graphical user interface simultaneously, with each display window providing an interface to a different application or to a different instance of the same application. A cursor in the graphical user interface may be manipulated by a user through the pointing device. The position of the cursor may be changed and / or an event, such as clicking a mouse button, generated to actuate a desired response.

[0024] One of various commercial operating systems, such as a version of Microsoft Windows™, a product of Microsoft Corporation located in Redmond, Wash, may be employed if suitably modified. The operating system is modified or created in accordance with the present disclosure as described.

[0025] LAN / WAN / Wireless adapter 112 can be connected to a network 130 (not a part of computer system 100), which can be any public or private computer system network or combination of networks, as known to those of skill in the art, including the Internet. Computer system 100 can communicate over network 130 with server system 140, which is also not pail of computer system 100, but can be implemented, for example, as a separate computer system 100.

[0026] Disclosed embodiments can be implemented using WebGPU technology as described in the WebGPU Working Draft published at www.w3.org / TR / webgpu / , which is a middleware API that drives low-level GPU APIs such as Vulkan, DirectX12 or Metal. WebGPU describes an API in which hardware and system specific instructions are handled by the API. An application built with WebGPU can virtually run on any kind of operating system and is hardware agnostic with respect to graphics processing units (GPUs). A WebGPU application can be accessed from a web browser, if compiled with WebAssembly support, or run natively on any system when compiled as an executable.

[0027] FIG. 2 illustrates a block diagram of elements of a graphics processing unit 200 in accordance with disclosed embodiments, configured to interact with a host computersystem 100, such as in a graphics adapter 110. In this example, host computer system 100 communicates with processing pipeline 250 of GPU 200. Processing pipeline 250 can include multiple physical GPU processing units, each of which can include multiple GPU cores, each of which can be used, individually or in combination, to implement the various processing units described below.

[0028] Processing pipeline 250 can include a vertex processing unit 204 that may have multiple vertex processors and can communicate with a vertex buffer 220 in GPU memory 214.

[0029] Processing pipeline 250 can include a texture and fragment processing unit 206 that may have multiple texture / fragment processors and can communicate with a texture buffer 218 in GPU memory 214.

[0030] Processing pipeline 250 can include a Z-compare (depth comparison) and blend processing unit 208 that may have multiple depth / blend processors and can communicate with GPU memory 214.

[0031] Processing pipeline 250 can include a frame buffer 210 which receives the processed graphics to be displayed and is read by host computer system 100.

[0032] GPU 200 includes a memory 214 that can be controlled by GPU memory manager 212. GPU memory 214 can include such elements as uniform buffer 216, texture buffer 218, vertex buffer 220, storage buffer 222, and general graphic memory 224. While this block diagram illustrates memory 214 as a unitary memory, those of skill in the art will recognize that certain elements shown as part of GPU memory 214, including specifically the various buffers, may be implemented as separate memory units associated with the corresponding processing units described herein.

[0033] Uniform buffer 216, in various embodiments, is a read-only data structure that is on GPU memory 214. Uniform buffer 216 can be prepared from the host computer system 100 by the user or the API. Uniform buffer 216 can be used as described herein.

[0034] Texture buffer 218, in various embodiments, is a specific memory structure on GPU memory 214 that contains RGB A (Red, Green, Blue, Alpha) information for colors. Texture buffer 218 can be used as described herein.

[0035] Vertex buffer 220, in various embodiments, is a specific memory structure on GPU memory 214 that contains geometrical information of a 3D object, such as vertex location, vertex indices to define triangles, normal of faces, etc. Vertex buffer 220 can be used as described herein.

[0036] Storage buffer 222, in various embodiments, is a generic part of the GPU memory 214 that can be used to read and write binary data. Similarly, general memory 224 refers to portions of GPU memory 214 that are generally available and usable by the GPU 200 or host computer system 100 via API.

[0037] Compute shader 226, in various embodiments, is a module that executes code on the GPU to perform mathematical instructions directly on the various buffers of GPU memory 214. A compute shader 226 can be written, for example, in WebGSL Shading Language (WGSL). A compute shader 226 can be executed by dispatching groups of threads on GPU cores, which run concurrently. The dispatch operation is complete when all threads have completed their execution.

[0038] In order to enable real-time design, visualization and simulation, disclosed embodiments include a computer-implemented software process that manages data residing in GPU memory buffers and going through compute and graphics pipelines.

[0039] FIG. 3 illustrates architecture 300 and process for real-time design, visualization, and simulation of a physical environment, such as an industrial system, in accordance with disclosed embodiments. The architecture and process illustrated in FIG. 2 can be implemented, for example, by the GPU 200 of FIG. 2 and within the context of a computer system 100 of FIG. 1.

[0040] As illustrated in FIG. 3, architecture 300 includes a compute pipeline 302 and a graphic s / rendering pipeline 304 running in parallel.

[0041] On the left of FIG. 2, the compute pipeline 302 controls the evolution of the physics-based simulation. On the right, the graphics pipeline 304 is responsible for displaying any update coming from the compute pipeline 302.

[0042] Each pipeline is controlled by its own time-stepping procedure. As discussed herein, compute pipeline 302 operates at a time step shown by simulation time loop Compute, where teach step n of time tcompute is of the durationGraphics pipeline 304 operates at a time step shown by simulation time loop graphics, where teach step n of time trophies is of the duration

[0043] For the graphics pipeline 304, the ^graphics time-step is typically the framerate when a picture is refreshed on the screen monitor. Typically, modem graphics engines aim to maintain 60 frames per second (FPS).

[0044] If ^compute = graphics, it means that each new drawing of a frame is synchronized with the new update of the physics-based solver running on the compute pipeline 302. If conVute > ^graphics, it means that the physics-based solution will be updated slower than the refresh framerate. It docs not imply that the graphics pipeline 304 will be stuck or lag; graphics pipeline 304 will still run smoothly at a target of 60 FPS, however any update on the geometry or visual post-processing of simulation results with get delayed.

[0045] The process illustrated in FIG. 3 begins at block 306. In this example, a uniform buffer, such as uniform buffer 216 of FIG. 2, contains the parameters that can be changed by the user. These parameters can be updated at run-time at any time; their definition is static but not their contained value. Note that the term “uniform buffer” is terminology employed by WebGPU to define such data-structure, which are read-only from the GPU side, but equivalent structures may be used.

[0046] The parameters from the uniform buffer are sent to (or read by) the simulation compute shader 308, which contains the code to solve physics-based problems, as well as sent to the graphics engine API 310 to program a vertex buffer 312 than only contains geometrical information for display. Note that vertex buffer 312, in this exemplary figure,is shown as vertex buffer 312a and vertex buffer 312b, which represent different time points in the process. Data in vertex buffer at 312a can be referred to as “first vertex buffer contents” and data in vertex buffer at 312b can be referred to as “second vertex buffer contents”. This geometrical information can also be sent from the vertex buffer 312 to the simulation compute shader 308 to provide data for the physics-based simulation. This can include, for example, the geometry of the computational domain where equations are solved.

[0047] The simulation compute shader 308 executes a physics-based solver to perform a simulation. This solver can be written, for example, in the WGSL shader language. The simulation compute shader 308 employs storage buffer 314 to save the data of the simulation at different time-steps. Note that storage buffer 314, in this exemplary figure, is shown as storage buffer 314a and storage buffer 314b, which represent different time points in the process, and are used for the time-integration of the physics-based solver. Data in storage buffer 314 at 314a can be referred to as “first storage buffer contents” and data in storage buffer 314 at 314b can be referred to as “second storage buffer contents”.

[0048] Once a compute process is completed, in other terms when the dispatch of the Compute Shader on the GPU is complete, the storage buffer 314b in GPU memory is read and the data can be directly used as second vertex buffer contents in vertex buffer 312b in the graphics pipeline 304. The vertex shader 312 in the graphics pipeline 304 will automatically interpret the new geometry change and will draw a new 3D object on the screen at the new frame. The output of the vertex buffer 314 and / or the texture buffer 318 can be written to a frame buffer for subsequent display by the computer system.

[0049] Concurrently, the data from storage buffer 314b can be sent to another a postprocessing compute shader 316 that will contain code for post-processing. This postprocessing can involve the preparation of colors to visualize and interpret simulation results. After the dispatch of the post-processing compute shader 316 is complete, the data are directly read in a texture buffer 318 in the graphics pipeline 304, so that in the next frame the colors of the 3D object will be updated. Note that texture buffer 318, in this exemplary figure, is shown as texture buffer 318a and texture buffer 318b, which representdifferent time steps in the processes described herein. Data in texture buffer 318 at 318a can be referred to as “first texture buffer contents” and data in texture buffer 318 at 318b can be referred to as “second texture buffer contents”.

[0050] The process and architecture as described herein provides significant technical advantages over current simulation techniques by unifying the simulation, performed by the compute pipeline 302 and the visualization performed by graphics pipeline 304 so that simulation and visualization can be performed in real time, which can include user interactions. This is contrasted, for example, with current systems in which the simulation cannot be performed in real time, because the steps of preparation, execution, and postprocessing are separated and often done by multiple different software applications. In such other systems, visualization and post-processing procedures only take care of graphics representation of results and are independent of the simulation processes.

[0051] FIG. 4 depicts a flowchart of a process 400 in accordance with disclosed embodiments that may be performed, for example, by a GPU of a computer system as disclosed herein. The process of FIG. 4 illustrates a single combined simulation iteration that may be performed in an architecture and process as illustrated in FIG. 3. It will be understood that each of the processes described below can be performed using GPU processors and cores as described above with respect to FIG. 2.

[0052] At 402, the GPU reads simulation parameters from a uniform buffer for processing by both a compute pipeline of the GPU and by a graphics pipeline of the GPU. The GPU can execute the compute pipeline and the graphics pipeline independently of each other with respect to time steps for performing the steps of this process.

[0053] At 404, the based on the parameters, the GPU programs a vertex buffer in the graphics pipeline to produce first vertex buffer contents stored in the vertex buffer.

[0054] At 406, based on the parameters, the first vertex buffer contents, and first storage buffer content in the compute pipeline, the GPU performs a simulation in the compute pipeline. The simulation can be, in some cases, a physics-based simulation and, inparticular, a physics-based simulation of an industrial operation. The first vertex buffer contents can be used as geometry inputs to the simulation. The simulation can be performed by a compute shader unit.

[0055] At 408, the GPU stores the results of the simulation in the storage buffer as second storage buffer contents. Here, the GPU writes to the storage buffer based on the results of the simulation.

[0056] At 410, the GPU stores second vertex buffer contents in the vertex buffer based on the second storage buffer contents. Here, the GPU writes to the vertex buffer based on the second storage buffer contents and / or based on first vertex buffer contents.

[0057] At 412, the GPU can perform post-processing, in the compute pipeline, based on the second vertex buffer contents and to produce texture data stored in a texture buffer of the graphics pipeline. This data can be stored in the texture buffer as second texture buffer contents.

[0058] At 414, the GPU writes data, in the graphics pipeline, from the vertex buffer to a frame buffer for display to a user. This step can also include writing data, in the graphics pipeline, from the texture buffer to the frame buffer for display to a user.

[0059] The computer system can thereafter display a real-time simulation output to a user based on the data stored in the frame buffer.

[0060] Of course, those of skill in the art will recognize that, unless specifically indicated or required by the sequence of operations, certain steps in the processes described above may be omitted, performed concurrently or sequentially, or performed in a different order.

[0061] Those skilled in the art will recognize that, for simplicity and clarity, the full structure and operation of all computer systems suitable for use with the present disclosure is not being depicted or described herein. Instead, only so much of a computer system as is unique to the present disclosure or necessary for an understanding of the present disclosure is depicted and described. The remainder of the construction and operation ofcomputer system 100 may conform to any of the various current implementations and practices known in the art.

[0062] It is important to note that while the disclosure includes a description in the context of a fully functional system, those skilled in the ail will appreciate that at least portions of the mechanism of the present disclosure are capable of being distributed in the form of instructions contained within a machine-usable, computer-usable, or computer-readable medium in any of a variety of forms, and that the present disclosure applies equally regardless of the particular type of instruction or signal bearing medium or storage medium utilized to actually carry out the distribution. Examples of machine usable / readable or computer usable / readable mediums include: nonvolatile, hard-coded type mediums such as read only memories (ROMs) or erasable, electrically programmable read only memories (EEPROMs), and user-recordable type mediums such as floppy disks, hard disk drives and compact disk read only memories (CD-ROMs) or digital versatile disks (DVDs).

[0063] Although an exemplary embodiment of the present disclosure has been described in detail, those skilled in the art will understand that various changes, substitutions, variations, and improvements disclosed herein may be made without departing from the spirit and scope of the disclosure in its broadest form.

[0064] None of the description in the present application should be read as implying that any particular element, step, or function is an essential element which must be included in the claim scope: the scope of patented subject matter is defined only by the allowed claims. Moreover, none of these claims are intended to invoke 35 USC § 112(f) unless the exact words "means for" are followed by a participle. The use of terms such as (but not limited to) “mechanism,” “module,” “device,” “unit,” “component,” “element,” “member,” “apparatus,” “machine,” “system,” “processor,” or “controller,” within a claim is understood and intended to refer to structures known to those skilled in the relevant art, as further modified or enhanced by the features of the claims themselves, and is not intended to invoke 35 U.S.C. § 112(f).

Claims

WHAT IS CLAIMED IS:

1. A method (400) for real-time simulation of a physical environment, the method performed by a graphical processing unit (GPU) (200) of a computer system (100) and comprising: reading (402) simulation parameters from a uniform buffer (216) for processing by a compute pipeline (302) of a GPU (200) of a computer system (100) and by a graphics pipeline (304) of the GPU (200); programming (404) a vertex buffer (220) in the graphics pipeline (304), based on the simulation parameters, to produce first vertex buffer contents; performing a simulation (406) in the compute pipeline (302), based on the parameters, the first vertex buffer contents, and first storage buffer contents to produce second storage buffer contents; storing (408) the second storage buffer content in a storage buffer (222); writing (410) second vertex buffer contents to the vertex buffer (220) based on the second storage buffer contents; and writing (414) data to a frame buffer (210) for display to a user, in the graphics pipeline (304), based on the second vertex buffer contents.

2. The method of claim 1, further comprising performing post processing (412), in the compute pipeline (302), based on the second vertex buffer contents to produce texture data, and wherein writing data to the frame buffer (210) is also based on the texture data.

3. The method of claim 1, wherein the simulation is a physics-based simulation of a physical environment.

4. The method of claim 1, wherein the simulation is a physics-based simulation of an industrial operation.

5. The method of claim 1, wherein the GPU (200) executes the compute pipeline (302) and the graphics pipeline (304) independently of each other with respect to time steps.

6. The method of claim 1, wherein the performing the simulation uses the first vertex buffer contents as geometry inputs.

7. The method of claim 1, wherein the simulation is performed by a compute shader unit (226).

8. A computer system (100) comprising: a processor (102); an accessible memory (108); and a graphics processing unit (GPU) (200), wherein the GPU (200) is configured to perform a process (400) as in any of claims 1-7.

9. A graphics processing unit (GPU) (200), comprising a GPU processing unit (250) having at least one GPU processor with multiple cores, wherein the GPU (200) is configured to perform a process (400) as in any of claims 1-7.

10. A non-transitory computer-readable medium (126) encoded with executable instructions that, when executed, cause one or more graphics processing units (GPUs) (200) to perform a process (400) as in any of claims 1-7.

Images