Convert discrete light attenuation to spectral data for rendering an object volume

Abstract

Apparatus, systems, and techniques are disclosed for rendering an image depicting light interacting with a medium having volume attenuation using spectral rendering of the medium in an optimized analog rendering three-color rendering scheme.

Description

Technical Field

[0001] At least one embodiment relates to computing techniques for performing and facilitating graphics applications. For example, at least one embodiment relates to operations used in rendering realistic images of complex scenes, where the complex scenes involve the transmission and reflection of light interacting with absorption and scattering media. Background Art

[0002] Image rendering (image synthesis) is the process of generating an image from scene data, which can involve various two-dimensional and / or three-dimensional models. The scene data can include the positions and geometries (shapes, sizes, orientations) of various objects in the scene, the positions of light sources, information about the intensity, type, and color of the light generated by the light sources, and information about the reflectivity / absorptivity of the objects. Rendering can then determine how each object in the scene should be presented to a viewer who is viewing the scene from a specific vantage point. Successful rendering algorithms typically involve physical laws, the physiological science of color perception, mathematics, and statistical modeling, in addition to complex software development and the effective utilization of processing hardware. Rendering can be static and / or dynamic. In the latter case, at least some of the objects and / or the viewing vantage points can move, such that at least some of the rendering is performed in real time. Brief Description of the Drawings

[0003] Figure 1A is a block diagram of an example computer system that optimizes spectral rendering of light interacting with a medium having volume attenuation using upsampling and an effective attenuation function, according to at least some embodiments;

[0004] Figure 1B is an example computing device that can implement upsampling and effective spectral rendering of light interacting with a medium having volume attenuation, according to at least one embodiment;

[0005] Figure 2 illustrates an example data flow that can be used to obtain an effective attenuation function for optimizing spectral rendering of light interacting with a medium having volume attenuation, according to at least some embodiments;

[0006] Figure 3A and Figure 3B illustrates an example of a possible form of an attenuation function that can be used to optimize spectral rendering of light interacting with a medium, according to at least some embodiments; Figure 3A depicts an example piecewise-constant function of the wavelength of light; Figure 3B depicts an example continuous function of the wavelength of light;

[0007] Figure 4A and Figure 4B illustrates an example evaluation scheme for evaluating the rendering accuracy of light interaction using an optimized attenuation function, according to at least some embodiments;Figure 4A An example evaluation scheme for evaluating the accuracy of rendering that uses an optimized attenuation function to interact light with a volume medium according to at least some embodiments is shown. Figure 4B An optimization of an attenuation function using primary colors in target space according to at least some embodiments is shown. Figure 4C An optimization 450 of an attenuation function using a combination of primary colors in target space according to at least some embodiments is shown.

[0008] Figure 5 A flowchart of an example method for effective spectral rendering of light interacting with a volume medium according to some embodiments of the present disclosure is shown.

[0009] Figure 6 A flowchart of an example method for optimizing an attenuation function for spectral rendering of light interacting with a volume medium according to some embodiments of the present disclosure is shown.

[0010] Figure 7A An inference and / or training logic according to at least one embodiment is shown.

[0011] Figure 7B An inference and / or training logic according to at least one embodiment is shown.

[0012] Figure 8 A training and deployment of a neural network according to at least one embodiment is shown.

[0013] Figure 9 An example data flow diagram of an advanced computing pipeline according to at least one embodiment is shown.

[0014] Figure 10 A system diagram of an example system for training, tuning, instantiating, and deploying a machine learning model in an advanced computing pipeline according to at least one embodiment is shown. Detailed Description

[0015] High-quality image rendering is a computationally intensive technology that involves a large amount of processing and memory resources. Therefore, optimizing algorithms and effectively implementing these algorithms on the hardware available to the user (developer) is an important technical goal. In addition, human visual perception is a complex interaction of the physics of light propagation and the interaction of light with material objects, as well as the physiological and neurological laws of photoreceptors in the human eye perceiving light, transmitting signals through the optic nerve, and neural processing by the human brain. Although the physical properties of light are determined by its spectral distribution (describing the representation of various wavelengths / frequencies in light), the perception of color depends on the presence of three types of color receptors (cones), whose spectral sensitivities are loosely concentrated in the red, green, and blue parts near the visible spectrum (380–750 nm). As a result, multiple spectral distributions of light may be perceived by the human eye as having the same color.

[0016] Color rendering in computer applications (and also in early television technology) is typically performed using three primary colors - red, green, and blue (RGB) - inspired by the approximation to human perception. The RGB rendering scheme involves specifying the amount of each primary color (R, G, B) in certain units (usually, each color ranges from [0, 255]). The spectral intensity I(λ) can be explicitly converted to the RGB scheme by projecting onto three RGB color matching functions c j (λ) (e.g., by calculating ∫dλI(λ)c j (λ) for each j = R, G, B and normalizing to the range [0, 255]). The inverse process of identifying the underlying spectral intensity I(λ) for R, G, B → I(λ) has multiple solutions (in fact, an infinite number). There are various upsampling procedures that can identify feasible solutions that well represent the spectral distribution in a particular (e.g., natural light) environment mapped to the target RGB values. Such upsampling may be beneficial for generating images that are superior to those rendered in the RGB scheme. For example, spectral rendering schemes may involve realistic models of light and light-matter interactions. Such models can operate on the light intensity I(λ) specified in the spectral (wavelength or equivalently, frequency) domain (e.g., as input and output). On the other hand, user data (such as sample images) is often provided in the RGB format.

[0017] When generating scene images that include volume media (e.g., haze, fog, dust, water, clouds, with a large atmospheric depth of air), RGB (or other trichromatic) rendering may result in different outcomes compared to the output of spectral rendering. This occurs because the actual particles of such media scatter and absorb light of different wavelengths to varying degrees. As a result, the intensity of light passing through the volume media decreases with the distance L through the media, I(L) = I(0)e -μ(λ)L , with an attenuation (extinction) coefficient μ(λ) depending on the wavelength λ. During propagation (or reflection) through an actual physical medium, monochromatic light remains monochromatic with the same wavelength (at least as long as emission and various non-linear effects remain small). On the other hand, the same monochromatic light rendered in a trichromatic scheme does change its color appearance because the attenuation of each underlying trichromatic component is different. For example, the blue part of the visible light range typically experiences stronger scattering than the red part, and absorption may be more pronounced for specific wavelength bands determined by the atomic and molecular properties of the medium. Therefore, monochromatic light rendered using the RGB (or other trichromatic) scheme may experience a different color shift during transmission through a volume medium (or when reflected from the same medium) compared to the color shift rendered in the spectral representation.

[0018] Aspects and embodiments of the present disclosure address these and other technical challenges by disclosing methods and systems capable of spectral rendering of images depicting volumetric media, which closely match the rendering of similar media in rendering schemes utilizing several (e.g., three, four, etc.) primary colors, including but not limited to the RGB color space (e.g., CIE RGB scheme), the CIE XYZ color space, the Academy Color Encoding System (ACES), the Rec. 2020 / BT. 2020 color space, or any other color encoding space. The disclosed embodiments enable fast and computationally efficient real-time rendering that can be used on a wide range of hardware platforms. In at least one embodiment, an attenuation function can be defined that describes the color dynamics when transmitting through and / or reflecting from a particular volumetric media and is parameterized by a finite number of fitting parameters (e.g., two, five, or any other number). The fitting parameters can be determined by an optimization process that minimizes the difference between (i) the appearance of transmitted / reflected light rendered in a spectral color rendering scheme and (ii) the appearance of the same light in a target color rendering scheme (e.g., RGB, XYZ, etc.).

[0019] Although medical imaging examples are used throughout the present disclosure to illustrate various concepts, substantially the same or similar concepts can be used for object recognition in other contexts, such as object recognition in driving or industrial environments, object recognition in security applications, scientific and investigative research, and many other applications.

[0020] System Architecture

[0021] Figure 1A is a block diagram of an example computer system 100 according to at least some embodiments, which uses upsampling and an effective attenuation function to optimize the spectral rendering of light interacting with a medium having volumetric attenuation. As Figure 1A shown, the computer system 100 for processing image data 166 can include an image processing server 101, an application server 160, and a client device 140 connected via a network 150. The network 150 can be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wireless network, a personal area network (PAN), or a combination thereof.

[0022] The image processing server 101 can be (or include) a desktop computer, a laptop computer, a smart phone, a tablet computer, a local server, a cloud server, a dedicated graphics server, a collection of multiple computing devices, a distributed computing system, a smart TV, an augmented reality device, or any other suitable computing device (or collection of computing devices) capable of performing the techniques described herein. The application server 160 and the client device 140 can similarly include any of the computing devices mentioned above. Alternatively, the client device 140 can be a computing device that lacks substantial computing resources but is capable of accessing and communicating with the image processing server 101 and / or the application server 160. The client device 140 can have a graphical user interface (GUI) 142 to facilitate user interaction with the client device 140, the application server 160, and the image processing server 101. The GUI 142 can be implemented on a desktop or laptop computer screen, a touch screen, a smart TV screen, or any combination thereof. The GUI 142 can include any pointing device (mouse, touchpad, stylus, finger, eye movement control device, etc.), keyboard, control bar, game console, etc. The GUI 142 can display static and moving objects, light sources, shadows, colors, menus, etc., for example, using a browser, a desktop application, a mobile application, etc.

[0023] The client device 140 can include a graphics application client 144 and an image processing client 146. The graphics application client 144 can be a client application provided and / or supported by a graphics application 162 running on the application server 160. The graphics application 162 can be any art, medical, scientific, engineering application, etc. For example, the graphics application 162 can be an image production application, a movie production application, a video game application, an engineering application, an architectural application, a flight simulation application, a scene reconstruction application, etc. The graphics application 162 can operate with an image data generator 164. The image data generator 164 can receive settings for one or more images from the graphics application 162. For example, the graphics application 162 can be a video application that provides a game context, such as the current position of a player relative to a building map. The image data generator 164 can generate coordinates of various objects, characteristics of the surfaces of these objects (e.g., reflectivity in various directions of light incidence and reflection), positions, brightness, and colors of light emitted by various light sources, etc.

[0024] The generated image data 166 can be provided (e.g., via network 150) to the image rendering engine 104 of the image processing server 101 for generating an image based on the image data 166. In some embodiments, the image data 166 can include one or more sample images to be used as a base image for generating additional images similar to the base image. For example, the image data 166 can include a number of outdoor and / or indoor images of an environment to be used as the landscape of a computer game (e.g., developed by a user of the graphics application 162 via the graphics application client 144). The task of the image processing server 101 can be to generate various images to support the computer game. The image processing server 101 can identify various objects and light sources in the received images and use this data to generate additional images involving different viewpoints, different arrangements of objects, some additional objects, different light sources, etc.

[0025] The generated image can be in any digital (e.g., pixel-based or vector-based) format, including but not limited to JPEG, GIF, PNG, BMP, TIFF, CIB, DIMAP, NITF, etc. In some embodiments, the image data 166 can be provided in a color space using a limited number of primary colors. For simplicity, the color space specified in the image data 166 is sometimes referred to herein as the RGB space, but it should be understood that any three-color space or other color space can alternatively be used. The image rendering engine 104 can include a color space to spectrum conversion component 106 to convert the color and intensity of the various light sources specified in the image data 166 into a spectral representation: R, G, B → I(λ). In some embodiments, the conversion can be performed differently depending on the type of the scene and / or the illumination of the scene. For example, if an object is illuminated by outdoor natural light, a first type of conversion of the light reflected from the specific object can be used, while when the object is illuminated by an indoor floodlight, a second (and different) type of conversion can be used. If the same object is illuminated by a street lamp at night, yet another different third type of conversion can be used.

[0026] The volume medial rendering engine 108 can determine how the converted spectral intensity I(λ) of the light source changes as the light travels through a (partially) transparent but attenuating medium. For example, the volume medial rendering engine 108 can determine from the image data 166 that the medium is fog of a specific density. The volume medial rendering engine 108 can access stored (e.g., in the memory 120) light absorption and scattering data and calculate the physical attenuation function μ(λ) of the fog based on the fog density and the accessed data. The volume medial rendering engine 108 can then determine a model attenuation function that ensures the consistency of the spectral rendering with the rendering of the same fog in the RGB space. In some embodiments, instead of specifying the type of the substance, the image data 166 can explicitly provide the attenuation coefficients μ for the primary RGB colorsj and the volume media rendering engine 108 can then determine the model attenuation function μ(λ) based on maximizing the similarity between the color appearance of the transmitted / reflected light in the spectral rendering scheme and the RGB rendering scheme.

[0027] The image rendering engine 104 can then use the determined model attenuation function μ(λ) to generate an output image based on the image data 166. Additionally, as depicted by the corresponding dashed box, the determined attenuation function 124 can be stored in the memory 120 for reuse in subsequent images (including images generated based on image data provided by applications other than the graphics application 162). In some embodiments, the operation of the image rendering engine 104 can be fully automated. In some embodiments, some operations can be controlled by the user via the GUI 109. If the user remotely accesses the image processing server 101 from the client device 140, at least a portion of the GUI 109 can be provided to the client device 140 as the GUI 142. The image processing client 146 can further facilitate the interaction between the user of the client device 140 and the image rendering engine 104. Some operations of the image rendering engine 104 that can be controlled via the image processing client 146 can include: selecting multiple fitting parameters for the model attenuation function μ(λ), selecting the type of target color space that the model attenuation function μ(λ) is intended to simulate, changing the metric used to evaluate the accuracy of the model attenuation function μ(λ) approximating the light attenuation in the target color space, etc.

[0028] The memory 120 can be communicatively coupled to one or more processing devices of the image processing server 101, such as one or more graphics processing units (GPUs) 110 and one or more central processing units (CPUs) 130. The image rendering engine 104 can be executed by the GPU 110 and / or the CPU 130 or a combination thereof. The image processing server 101 can further include various input / output (I / O) components 134 to facilitate the exchange of information with various peripheral devices.

[0029] Although the image processing server 101, the application server 160, and the client device 140 are shown as separate devices in Figure 1A , in various embodiments, any two (or all) of these devices can be combined on a single computing machine. For example, the image processing server 101 and the application server 160 can be executed on the same machine remotely accessed by the client device 140. In another embodiment, the image processing server 101, the application server 160, and the client device 140 can be executed on the user's (or developer's) computer (e.g., a desktop or laptop).

[0030] Figure 1BAn example computing device 102 according to at least one embodiment that can implement upsampling of light interacting with a volumetrically attenuating medium and effective spectral rendering. In some embodiments, the computing device 102 can be an image processing server 101 or another computing device implementing an image rendering engine 104. In some embodiments, the image rendering engine 104 can be executed by one or more GPUs 110 and can include a color space to spectral conversion component 106 and a volumetric medium rendering engine 108 that perform spectral upsampling and simulation of volumetric media in a spectral representation. In some embodiments, the GPU 110 includes a plurality of cores 111, each core capable of executing multiple threads 112. Each core can run multiple threads 112 simultaneously (e.g., in parallel). In some embodiments, the threads 112 can access registers 113. The registers 113 can be thread-specific registers, and access to the registers is limited to the corresponding thread. Additionally, shared registers 114 can be accessed by all threads of the core. In at least one embodiment, each core 111 can include a scheduler 115 to allocate computing tasks and processes among different threads 112 of the core 111. A dispatch unit 116 can implement the scheduled tasks on the appropriate threads using the correct private registers 113 and shared registers 114. The computing device 102 can include one or more input / output components 134 to facilitate information exchange with peripheral devices and users and developers.

[0031] In some embodiments, the GPU 110 can have a (high-speed) cache 118, and access to the cache 118 can be shared by multiple cores 111. Additionally, the computing device 102 can include a GPU memory 119, where the GPU 110 can store intermediate and / or final results (outputs) of various computations performed by the GPU 110. After completing a particular task, the GPU 110 (or CPU 130) can move the output to the (main) memory 132. In some embodiments, the CPU 130 can perform tasks involving serial computations (e.g., optimization of attenuation functions), while the GPU 110 can perform tasks suitable for parallel processing (e.g., rendering an image using the optimized attenuation function). In some embodiments, the image rendering engine 104 can determine which processes will be executed on the GPU 110 and which processes will be executed on the CPU 130. In other embodiments, the CPU 130 can determine which processes will be executed on the GPU 110 and which processes will be executed on the CPU 130.

[0032] Figure 2FIG. 200 shows an example data stream 200 according to at least some embodiments, which can be used to obtain an effective attenuation function for optimizing the spectral rendering of light interacting with a medium having volume attenuation. As schematically depicted, input data 202 can be provided to volume propagation modeling 210. The input data 202 can include a description of one or more individual volume media present in the scene being rendered. The scene can be associated with an outdoor or indoor environment having any number of objects and light sources, such as emissive light sources (the sun, light bulbs / light emitting diodes, natural or industrial fires, etc.), reflective light sources (e.g., mirrors, surfaces of liquid substances, smooth surfaces, matte surfaces, etc.), diffuse / scattering light sources (e.g., the atmosphere, particulate matter, etc.). Some objects may be light sources simultaneously. The input data 202 can include the positions (e.g., coordinates in image space), sizes, orientations, and properties of various objects and light sources. The characteristics of a light source can include its radiance, direction, and a characterization of the color of the emitted (or reflected / scattered) light. The properties of an object may include characteristics of the object's surface, such as spectral reflectance, transmittance, and absorptance (e.g., bidirectional scattering function) for various incident and reflected / transmitted light directions. In some embodiments, the image to be rendered is part of a sequence of dynamic images (frames) in which the objects and / or light sources move relative to the viewer. For example, such an image sequence can correspond to a stationary observer observing a moving object, a moving observer observing a stationary object, or a moving observer observing a moving object.

[0033] The input data 202 can enable the image rendering engine 104 to render one or more images, which can include simulating how much light (and of what color) the various elements (e.g., pixels or voxels) of the image will receive from the various objects in the scene. As the input data 202 can specify, the scene can include objects and media, which are characterized by the volume attenuation of light interacting with the media, e.g., propagating, passing through (transmitting), reflecting, or scattering light from the media. The media can include any gaseous substance (e.g., the atmosphere), liquid substance (e.g., water, oil, beverages, etc.), at least partially light-transmissive solid substance (e.g., glass, plastic, ice, etc.), or any suspension, solution, or combination thereof (e.g., fog, mist, dust, etc.). Hereinafter, for the sake of brevity, a single medium is referred to, but it should be understood that multiple media present in the scene can be treated in a substantially similar manner. The input data 202 can indicate the intensity of the interaction of light with the media. For example, it can be characterized by an absorption coefficient μ a and a scattering coefficient μ s to characterize the medium. The absorption coefficient μ a specifies the fraction of light absorbed by the medium per unit distance of light propagation. Similarly, the scattering coefficient μ s specifies the fraction of light scattered by the medium per unit distance.

[0034] Due to absorption and scattering, the intensity of light may decrease exponentially with the propagation distance,

[0035] I(L) = I(0)e -μL ,

[0036] where μ is the extinction coefficient. In some embodiments, for example, in the case where multiple scattering of light can be ignored, the extinction coefficient can be approximated as the sum of the absorption coefficient and the scattering coefficient, μ = σ a + σ s . In some embodiments that consider multiple scattering of light, the extinction coefficient can depend non-linearly on the absorption and scattering coefficients. In one exemplary embodiment, as described below, the extinction coefficient can be approximated as In some embodiments, other forms of the attenuation coefficient μ can be used.

[0037] Attenuation (from scattering and / or absorption) generally depends on the wavelength of light (or equivalently, the frequency). For example, σ s (λ) may be greater at shorter wavelengths λ (e.g., blue light) than at longer wavelengths (e.g., red light). Similarly, a particular medium may have stronger absorption σ a (λ) at certain specific colors (e.g., red), and weaker absorption at other wavelengths. As a result, monochromatic light may attenuate as it passes through a distance L,

[0038] I(λ,L) = I(λ)e -μ(λ)L ,

[0039] The attenuation coefficient μ(λ) depends on the wavelength λ of light. When non-monochromatic light propagates through a medium (or otherwise interacts with the medium), the spectrum of the light may change because different spectral components may attenuate in different ways. Accordingly, the appearance of the incident light to a human observer before the light interacts with the medium may be different from the appearance of the transmitted light after the interaction with the medium (e.g., the transmitted light is perceived as having a different color). It should be understood that the exponential dependence of the intensity on the distance L is for illustrative purposes only. In some embodiments, the attenuation coefficient can be position-dependent, e.g., μ(λ,z). In such embodiments, the intensity of light can be determined by integrating the attenuation coefficient over the distance the light travels,

[0040]

[0041] Throughout this disclosure, for the sake of brevity and clarity, reference is made to simple exponential attenuation in a homogeneous volume medium, but substantially the same techniques can be used for heterogeneous media.

[0042] The input data 202 may include information that allows the capture of such color changes. For example, the input data may include an image of an actual scene (e.g., a digital photograph). The image may use a three-color space (e.g., RGB, XYZ, ACES, etc.) or some other target color space with N reference colors (e.g., red, green, blue, and / or some other colors). Each reference color j may be associated with a corresponding color matching function c min ,λ max defined in the wavelength interval [λ j (λ), and this interval may be the visible range (λ min = 380nm, λ max = 700nm)) or some other range. For example, in industrial, medical, and automotive (e.g., autonomous vehicle) applications, various other ranges of electromagnetic waves (e.g., infrared, ultraviolet, X-rays, etc.) may be used. The term "light" should be understood to cover all such relevant ranges of electromagnetic waves and is not meant to be limited to the visible wavelength range. Light with a given spectral intensity I(λ) can be represented in the target color space, and its values [a1, a2, …, a N are determined by projecting the spectral intensity I(λ) onto the corresponding color matching functions

[0043] The image in the input data 202 may have information about the color of the light incident on the volume medium (e.g., determined from a direct observation of a light source or other objects illuminated by light) and the color of the light transmitted through the medium. In some cases, the incident light may be known light (e.g., D65 natural light in the RGB color space). In some cases, the transmitted light may pass through a cloud of known (e.g., determinable from the image) thickness L. In some embodiments, the volume medium rendering engine 108 may identify that the incident light in the image has a representation [a1, a2, …, a N in the target space. The volume medium rendering engine 108 may further determine (e.g., based on the known or estimated medium thickness L) that the transmitted light has a representation [b1, b2, …, b N different from that of the incident light. The volume attenuation coefficient μ j of various primary colors j in the target space can be identified as μ j = L -1 ln(a j / b j ), such that each primary color is attenuated with its respective coefficient μ j , meaning that the transmitted light attenuates with distance:

[0044] Incident light: [a1, a2, …, a N

[0045] Transmitted light:​

[0046] For different primary colors, the attenuation coefficient μ j may be different. For example, in the RGB target color space, μ T ≠ μ G ≠ μ B . In some embodiments, the attenuation coefficient μ j may be provided together with the input data 202 instead of being determined from an existing image. For example, an application (e.g., the graphics application 162 or the image data generator 164) may explicitly specify the attenuation coefficients of various media to be rendered in the target space. To identify what attenuation function e -μ(λ)L should be used in the spectral representation to match the appearance and color of an image (present or being rendered) in the target color space, the volume media rendering engine 108 may perform some of the operations described below.

[0047] As schematically depicted by the volume propagation modeling 210 block, the spectral intensity I(λ) of the light 214 incident on the medium 212 may first be determined. In some embodiments, the spectral intensity I(λ) may be independently known. For example, natural light may be present in a daytime image of an outdoor scene. In some embodiments, I(λ) may be specified by the developer. In some embodiments, I(λ) may be determined from the image based on the color appearance of the objects in the image that are not obscured by the medium 212. In those cases where the spectral intensity I(λ) is not explicit (e.g., the input data 202 is not provided), and only the representation [a1, a2, …, a N in the target color space is available, the volume media rendering engine 108 may first select a transformation [a1, a2, …, a N → I(λ) from the possible spectral intensities that may have the correct representation [a1, a2, …, a N in the target color space.

[0048] The volume media rendering engine 108 may also identify how the primary colors attenuate with distance L (as shown in the schematic diagram 220), e.g., by identifying the corresponding attenuation coefficients μ j explicitly provided in the input data 202, determining the attenuation coefficients from the images included in the input data 202, accessing a database of various media properties, etc. Similar to how the attenuation coefficient μ j characterizes the transmitted light 216, additional reflection coefficients r j describing how the reflected light 218 is represented by the primary colors in the target color space may be obtained similarly, e.g., [r1·a1, r2·a2, …, r N ·a N. Although various embodiments have been described in the context of transmitted light for the sake of simplicity, substantially similar operations can be used to model the attenuation function of a volumetric medium, such as the spectral reflectance r(λ), to obtain maximum similarity with the reflected light rendered in the target space.

[0049] The volume medium rendering engine 108 can then use the attenuation function module 230 to define the attenuation function t(λ,L) in the form of a model function of the wavelength, which can relate the intensity of the transmitted light to the intensity of the incident light: I(λ,L) = t(λ,L)·I(λ). (Similarly, an attenuation function for reflection r(λ) can be defined, which relates the intensity of the reflected light to the intensity of the incident light: I R (λ) = R(λ)·I(λ).) The attenuation function can be modeled in various ways. Figure 3A and Figure 3B shows an example of a possible form of the attenuation function t(λ,L) that can be used to optimize the spectral rendering of light interacting with a medium. Figure 3A Depicts a piecewise-constant function 300 of the light wavelength. Figure 3B Depicts a continuous function 310 of the light wavelength. In some embodiments, as a function of the distance L, the attenuation function T(λ,L) can be continuous, such as an exponentially decreasing function of the distance L, even though a non-exponential (e.g., power-law) function of the distance L can be used. The attenuation function can be defined at least in the light interval of interest [λ min , λ max , for example, the visible light range or any other wavelength range. The attenuation function may depend on one or more parameters. In a particular non-limiting example, the attenuation function can be assumed to take N values in N regions separated by N - 1 boundaries λ α (e.g., the function 300 depicted in Figure 3A is used for N = 3).

[0050]

[0051] In one embodiment, the parameter can be taken to be equal to the attenuation coefficient μ in the target space j , while the set of boundaries {λ α} is determined by optimization. In another embodiment, the parameter is also determined by optimization, together with the boundaries {λ α}. In yet another embodiment, the parameter can be used to parameterize a continuous attenuation function (e.g., as depicted in Figure 3B ), such as a polynomial function, a piecewise-polynomial function, or any other function of λ. For example, the value can refer to the attenuation function (or its logarithm - L -1The average value of log t(λ,L)) in the interval [λ min , λ1] (similar to the value associated with the interval [λ1, λ2], and so on). For another example, the parameter can refer to the maximum value of the attenuation function in each interval (or its logarithm - L -1 log t(λ,L)) or the attenuation function (or its logarithm - L - 1 log t(λ,L)) at the midpoint of each interval (for example, the value may be related to the value of the attenuation function or its logarithm at (λ min + λ1) / 2). Almost an infinite number of different attenuation functions that embody various mappings can be used.

[0052] Returning to the reference Figure 2 , the volume rendering engine 108 can use the defined model attenuation function and the parameter λ α (and optionally, if subject to optimization) with some initial values to determine the spectral distribution of the transmitted light that can be calculated, I(λ,L) = t(λ,L)·I(λ). Next, the volume rendering engine 108 can project the calculated intensity I(λ,L) onto the color matching function c j (λ) to obtain the representation of the transmitted light [d1, d2, …, d N in the target color space,

[0053]

[0054] and perform a mapping 240 on the obtained representation to the target color space. As described in more detail below regarding Figure 4A -C, some evaluation metrics 250 can be used to evaluate how closely the attenuation function based on the initial parameters represents the transmitted light in the target color space. The optimization module 260 can then determine how to adjust the initial parameters of the attenuation function to improve the similarity between the two representations. After the attenuation function module 230 adjusts the parameters of the attenuation function, the process represented by the loop in Figure 2 can be repeated until the optimization module determines that the desired (e.g., predetermined) accuracy is achieved. The attenuation function (with the determined optimized parameters) can then be used for image rendering using the spectral representation. Such image rendering can be performed on the same image for which the incident light intensity I(λ) and the transmitted light intensity I(λ,L) have been used to optimize the attenuation function. In addition, the determined attenuation function can be used to render subsequent images with similar media and various other incident light intensities I1(λ), I2(λ)… etc.

[0055] Figure 4AAn example evaluation scheme 400 for evaluating the accuracy of the interaction between rendering light and a volume medium using an optimized attenuation function is shown, according to at least some embodiments. Four stages of the evaluation scheme 400 are schematically shown: selecting incident light 402, performing medium modeling 404, calculating transmitted light 406, and performing optimization 408. First, incident light 402 can be selected. The incident light can be defined in a three-color (or any other) target color space (e.g., XYZ or sRGB), e.g., by a set of three-color values {a j}. The incident light 402 can be a primary color of the target color space or any mixture of two or more primary colors of the target color space. The incident light 402 can be any light source, such as but not limited to white light, natural outdoor light, or any other combination of colors. The incident light 402 can also be represented by its spectral intensity I(λ) 420 (e.g., using various color upsampling techniques). Then the transmitted light 406 can be obtained in two representations. For example, using the known attenuation coefficients {μ j} of the individual primary colors in the volume medium 412, for a given length of the medium, the transmitted light 406 in the target color space can be obtained. Similarly, the transmitted light intensity I(λ)e -μ(λ)L in the spectral representation 424 can be obtained further using the modeled attenuation coefficient μ(λ) 422 (for the modeled volume medium 414), which may have one or more optimization parameters. In some embodiments, instead of the exponential attenuation function e -μ(λ)L , some other attenuation function t(λ,L) in a more general form can be used.

[0056] The transmitted light 406 in the target color space and the spectral representation can be characterized on an equal footing using a two-dimensional chromaticity plane in some color space (e.g., XYZ or sRGB), which can be the same or a different space from the target color. In some embodiments, the chromaticity plane xy can be the chromaticity plane of the CIE XYZ color space. As Figure 4A shown, the transmitted light 406 in the target color space can be mapped to chromaticity values (x, y), while the same transmitted light in the spectral representation can be mapped to chromaticity values (X, Y). The error measure 416 for the accuracy of the selected parameters characterizing the attenuation function can be some function f(X,Y;x,y) of the chromaticity values X, Y of the transmitted light in the spectral representation and the chromaticity values x, y of the transmitted light in the target color space. For example, the error measure can be based on the Euclidean distance between the respective coordinates, f(X,Y;x,y) = (X - x) 2 +(Y - y) 2。The error minimization module 418 can modify the parameters of the attenuation coefficient (or more generally, the attenuation function) using the determined error metric to reduce the error metric. Although in the above example, only the relative chromaticities (rather than the total intensities) of two transmitted lights were compared, in some embodiments, the total intensities of the transmitted lights can also be compared, f = (X - x) 2 +(Y - y) 2 +w(I - i) 2 , where I is the total intensity of the transmitted light in the spectral representation and i is the total intensity of the transmitted light in the target color space. The weight w can describe the importance of color perception relative to brightness perception and can be determined from empirical testing. Similarly, in some embodiments, a complete set of three (not normalized to unit luminance) color values (e.g., X, Y, Z) can be used. Then, the error measure 416 can be the sum of the Euclidean distances in the three-dimensional space of the chromaticity values. In some embodiments, the weights of the distances along each chromaticity axis can be different, f = α·(X - x) 2 +β·(Y - y) 2 +γ·(Z - z) 2 , with weights α, β, γ, at least some of which may not be equal. In some embodiments, the error measure 416 can be defined for different sets of incident lights, e.g.,

[0057]

[0058] where the index i enumerates the various incident lights. In some embodiments, the measurement f can weight the error along two (or three) axes using different functions, e.g., f = ∑ i [a·|X i - x i | k +b·|Y i - y i | l with different k ≠ l.

[0059] In one example, the three incident lights can be the primary colors of the target color space. Figure 4B Shows an attenuation function optimization 430 using the primary colors of the target space according to at least some embodiments. Figure 4B The inner part of the shaded region in corresponds to the various colors that the human eye can perceive. The chromaticity values x and y characterize the relative presence of the three primary colors. The white circle depicts the chromaticity of the light in the target color space, the black circle depicts the chromaticity of the light in the spectral representation, and the black / white circle depicts the light with the same chromaticity (perceived color) in both representations. Figure 4B Shown in are the three primary colors: the first primary color 431, the second primary color 432, and the third primary color 433. As Figure 4BAs shown, the same incident light (left figure) 431 in the two representations remains its color in the target space after transmission (right figure), but in the spectral representation, it transforms into a new color described by chromaticity values x and y, depicted by point 441. Similarly, in the spectral representation, the second (third) primary color moves from point 432 (point 433) to point 442 (point 443). The error measurement 416 for optimizing the attenuation function can be any of the distances 431 - 441, 432 - 442, and / or 433 - 443 alone or in combination (e.g., sum of distances, sum of squares of distances, weighted sum, etc.). The optimization can be performed until the error measurement 416 is minimized.

[0060] In another example, the incident light can be any combination of the primary colors of the target color space. As shown in the figure. Figure 4C Illustrated is an attenuation function optimization 450 using a combination of primary colors of the target space according to at least some embodiments. As shown in the figure. Figure 4C (Left figure) is the incident light having the same representation in the target color space and the spectral representation (as shown by the black / white circle 454). After transmission (right figure), the incident light moves to point 455 in the target color space (as shown by the corresponding white circle). In the spectral representation, the same transmitted light moves to a different point 456 (as shown by the corresponding black circle). The error measurement 416 for optimizing the attenuation function can be the distance 455 - 456. The optimization can be performed until the error measurement 416 is minimized. Although a single incident / transmitted light is depicted in Figure 4C , the optimization can be performed based on the sum of the error measurements of multiple incident / transmitted lights, as described in more detail above.

[0061] The optimization of the parameters of the attenuation function can be performed iteratively using any optimization algorithm, including gradient descent, finite difference, Newton's method, Hessian-based methods, golden section method, etc. After adjusting the parameters, the position of the point (X, Y) corresponding to the transmitted light in the spectral representation can be recalculated, and a new value of the error metric f can be determined. The optimization can be performed until one or more accuracy conditions are met, e.g., until the error metric f becomes less than a predetermined threshold f T . Alternatively, the optimization can be performed until two or more subsequent iterations fail to improve the error metric by at least a certain value Δf or a certain percentage of the metric (e.g., 5%, 10%, etc.).

[0062] In some embodiments, the parameters of the attenuation function can be determined using a single spectral distribution I(λ) of the incident light. In at least some embodiments, the optimization can involve multiple spectral distributions I k (λ). A separate error measurement f k can be calculated for each distribution I k (λ), and based on all the measurements f kPerform optimization. For example, a global error metric, F = ∑ k f k (or or some other global metric). The parameters of the attenuation function can then be determined based on the optimization of the global metric F of the error, substantially as described above. In some embodiments, the optimization can be performed for a single spectral distribution but multiple distances L. In some embodiments, the optimization can be performed based on multiple spectral distributions and multiple distances.

[0063] Although the above description of the identification of the optimized attenuation function relates to a specific (e.g., known) medium, substantially similar operations can be performed when a single attenuation function is used to render an image that may contain multiple media. For example, in some cases, the image to be rendered may have one or more media, and its specific attenuation coefficient (in the target color space) may be unknown. In such cases, a pre-computed general attenuation function can be used, which is optimized for rendering various volume media that may be encountered in similar types of images, such as fog, dust, haze, clouds, water, etc. Such a general attenuation function can be determined by optimizing the global error measurement F = ∑ k f k determined for multiple (e.g., m) volume media for which the attenuation coefficients are available. (In some embodiments, for a particular medium, only some of the attenuation coefficients may be known, e.g., known for the red and green primaries but unknown for blue.) Additionally, in such cases, the global error measurement can include several (e.g., n) different incident lights (a total of n x m different contributions to the global error measurement F = ∑ k f k ). The general attenuation function obtained in this way can then be used to render new volume media that were not used during the previous optimization.

[0064] The following table shows example boundaries λ obtained by optimization for various trichromatic spaces in the interval λ ∈ [380 nm, 780 nm], calculating the integral over wavelengths with 5 nm discretization. α using 5 nm discretization to calculate the integral over wavelengths.

[0065] Color space <![CDATA[λ1]]> <![CDATA[λ2]]> XYZ 500nm 575nm sRGB 485nm 595nm ACES 505nm 550nm ACEScg 505nm 570nm Rec2020 500nm 570nm

[0066] The values listed in this table were obtained by using a white light source and three different media, one for each primary color ( Figure 4B 431, 432, and 433), and as the error metric the sum of all three distances, 431 - 441, 432–442, and 433–443.

[0067] Although the above embodiments have described the attenuation of transmitted light, color matching can be performed in a similar manner for light reflected from a medium having both scattering and absorption. In at least one embodiment, the light I propagating in the forward direction T and the light I propagating in the reverse direction R can be modeled by the coupling equations:

[0068]

[0069]

[0070] where z is the distance the light beam travels; the term σ a I T and σ a I R describe the absorption of the corresponding light, while the term ±σ s (I R -I T ) describes the scattering effect. For example, due to scattering into the light I R propagating in the reverse direction, the intensity of the light I T propagating in the forward direction decreases, and due to the scattering of the light I R propagating in the backward direction, the intensity of the light I T propagating in the forward direction increases.

[0071] The above equation shows that the intensity of the forward-propagating light depends on the distance z as

[0072] I T (z) = I0e -μz ,

[0073] where each of the scattering coefficient σ s (λ), the absorption coefficient σ s (λ), and the attenuation coefficient μ(λ) can depend on the wavelength λ. Associating the constant I0 with the incident light, I0 = I(λ), and further taking the backward-propagating light I R (0) at z = 0 as the light reflected from the medium, the reflectivity r(λ) = I R (0) / I0 can be expressed as

[0074]

[0075] As described above, the reflectivity r(λ) in the spectral representation can be determined in a manner substantially similar to how the transmittance t(λ,L) is determined. Specifically, the modeled reflectivity can include one or more fitting parameters (e.g., reflectivity values within various spectral intervals separated by the boundary λ α ). )。Then, fitting parameters are determined based on the similarity between the color appearance of the reflected light in the target color space and the simulated reflected light in the spectral representation.

[0076] Figure 5 is a flowchart of an example method 500 for effective spectral rendering of light interacting with a volume medium according to some embodiments of the present disclosure. In some embodiments, method 500 may be executed by a processing unit of the image processing server 101 of FIG. 1 that executes instructions of one or more software modules (e.g., the image rendering engine 104). Method 500 may be executed by one or more processing units (e.g., a CPU and / or a GPU), which may include (or communicate with) one or more memory devices. In some embodiments, method 500 may be executed by multiple processing threads (e.g., CPU threads and / or GPU threads), each thread performing the operations of one or more individual functions, routines, subroutines, or methods. In some embodiments, the processing threads implementing method 500 may be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the processing threads implementing method 500 may execute asynchronously relative to each other. Compared with Figure 5 the order shown, the various operations of method 500 may be executed in a different order. Certain operations of the method may occur simultaneously with other operations. In some embodiments, Figure 5 one or more of the operations shown in

[0077] Method 500 may be executed to render an image using a spectral representation, where the image is based on image data available (e.g., provided by a user or an image developer) in a representation different from the spectral representation (e.g., in any trichromatic space). In some embodiments, the image data may include an input image in a trichromatic (e.g., RGB, XYZ, etc.) space, and the image rendering engine 104 may render a copy of the input image in the spectral representation while reproducing the colors of the input image with high precision. In some embodiments, the image rendering engine 104 may use the input image as a base image to render other images. For example, other rendered images may be images with different positions of the same object and light source, images with different objects and / or light sources, animated images, motion picture frames (e.g., video game images), etc. In block 510, the processing unit executing method 500 may identify a first light having a first representation in a first color rendering scheme (CRS) and a second representation in a second CRS. For example, the first light may be incident light identified in the input image or other input data. The first CRS may be a spectral rendering scheme, and the first representation, e.g., I(λ), may specify the light intensity as a function of the wavelength λ. The second CRS may be a trichromatic rendering scheme, where the same first light is represented by trichromatic values [a1, a2, a3]. In some embodiments, the second representation is provided with the input data (e.g., determined from the input image), while the first representation is simulated based on the trichromatic values. In some embodiments, more than three values may be used to identify the first light in the second RCS.

[0078] In block 520, method 500 may continue for the processing unit to identify an attenuation function characterizing the interaction of the first light with the medium (as described in more detail in conjunction with Figure 3A and Figure 3B ). The medium may be any volume medium that is at least partially transparent to light (e.g., the first light); when the first light interacts with the medium (e.g., by transmission or reflection or both), the medium may absorb and scatter the light, resulting in attenuation of the intensity of the first light. In some embodiments, the degree to which the first light is absorbed and / or scattered may be specified in the input data, e.g., by the scattering and / or absorption coefficients of the different primary colors of the second SRC. The identified attenuation function may describe the transmission of light through the volume medium, e.g., t(λ, L) or ln[t(λ, L)] in an instance of transmission or r(λ) in an instance of reflection. The identification of the attenuation function may include specifying the general form of the attenuation function, the number and type of fitting parameters, the wavelength range characterized by the attenuation function, etc. In some embodiments, the attenuation function may be an exponential function t(λ, L) = e -μ(λ)L .

[0079] In some embodiments, the attenuation function may be a piecewise constant function of wavelength that takes multiple values Each of the plurality of values is associated with a respective one of a plurality of wavelength intervals, such as [λ min , λ1], [λ1, λ2], … [λ N-1 , λ max . In some embodiments, one or more fitting parameters of the attenuation function may include at least one boundary λ j between adjacent intervals of the plurality of wavelength intervals. In some embodiments, the plurality of values (e.g., ) one or more of which may be based on the attenuation coefficients μ1, μ2, … μ N of the medium for the respective colors of the second CRS. In some cases, the attenuation coefficients μ1, μ2, … μ N of the medium may be specified in the input data or determined from the input image. In some embodiments, some of the values may be equal to (or proportional to) some of the attenuation coefficients μ1, μ2, … μ N . In some embodiments, M values may be different from N attenuation coefficients μ j . In some embodiments, M values may be the same as N attenuation coefficients μ j . In some embodiments, the values

[0080] may be selected as follows. At block 530, method 500 may continue for the processing unit to determine a third representation of the first light interacting with the medium in the first CRS using the attenuation function. For example, the third representation may be the spectral intensity of the first light transmitted through the medium, such as t(λ, L)I(λ), or the spectral intensity of the first light reflected from the medium, r(λ)I(λ). In addition to the third representation, a fourth representation characterizing how the first light interacts with the medium in the second CRS may be used. For example, the fourth representation may include the trichromatic values of the first light after the first light has passed through (or been reflected from) the medium.

[0081] At block 540, method 500 may continue for the processing unit to determine (e.g., as described in conjunction with Figure 4A and Figure 4B in more detail) one or more fitting parameters of the attenuation function, at least in part based on minimizing the difference between the third representation of the first light and the fourth representation of the first light interacting with the medium. In some embodiments, determining one or more fitting parameters may be performed in conjunction with Figure 6 the method 600 described.

[0082] At block 550, method 500 may continue to process the unit to render an image including at least one of a first light interacting with a medium or a second light interacting with the medium using an attenuation function and one or more determined fitting parameters. For example, the determined attenuation function may be used to render at least a portion of an input image with a first light in a spectral representation (first CRS). Specifically, the image rendering engine may use the spectral representation and the determined attenuation function to render a copy of the input image. In some embodiments, the image rendering engine may use the spectral representation and the determined attenuation function to create one or more additional images. The creation of such one or more additional images may include rendering one or more second (third, fourth, etc.) lights having spectral compositions (e.g., I2(λ), I3(λ), I4(λ),...) different from the spectral composition I(λ) of the first light used to obtain the attenuation function).

[0083] Figure 6 is a flowchart of an example method 600 for optimizing an attenuation function for spectral rendering of light interacting with a volume medium according to some embodiments of the present disclosure. In some embodiments, method 600 may be performed by the image rendering engine 104 as part of method 500 (e.g., in conjunction with block 540). More specifically, determining the fitting parameters may include identifying, at block 610, a plurality of chromaticity values (e.g., X and Y) corresponding to a third representation (e.g., t(λ,L)I(λ) or r(λ)I(λ)) of the first light. In one embodiment, the plurality of reference chromaticity values may be associated with a plurality of primary colors of a second CRS (e.g., XYZ color space or RGB color space). As shown in block 620 of method 600, determining the fitting parameters may further include determining a measure (e.g., error measure f) representing the distance between each of the plurality of chromaticity values and a corresponding one of the plurality of reference chromaticity values (e.g., (x j ,y j ), as described in more detail in conjunction with Figure 4A , Figure 4B and Figure 4C . As shown in block 620 of method 600, determining the fitting parameters may further include modifying at least some of the fitting parameters to improve (e.g., minimize) the determined measure. In some embodiments, the modification of the fitting parameters may be performed using multiple iterations.

[0084] Inference and training logic

[0085] Figure 7A illustrates inference and / or training logic 715 for performing inference and / or training operations associated with one or more embodiments.

[0086] In at least one embodiment, the inference and / or training logic 715 can include, but is not limited to, code and / or data storage 701 for storing forward and / or output weights and / or input / output data, and / or other parameters that configure neurons or layers of a neural network trained to and / or for inference in aspects of one or more embodiments. In at least one embodiment, the training logic 715 can include or be coupled to code and / or data storage 701 for storing graph code or other software to control timing and / or sequencing, where weight and / or other parameter information is loaded to configure the logic, including integer and / or floating point units (collectively arithmetic logic units (ALUs) or simple circuits). In at least one embodiment, the code (such as graph code) loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, the code and / or data storage 701 stores the weight parameters and / or input / output data of each layer of the neural network used in training or used during inference during forward propagation of the input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, any portion of the code and / or data storage 701 can be included within other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0087] In at least one embodiment, any portion of the code and / or data storage 701 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 701 can be cache memory, dynamic random access memory (“DRAM”), static random access memory (“SRAM”), non-volatile memory (such as flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage 701 is internal or external to the processor, e.g., or consists of DRAM, SRAM, flash memory, or some other storage type, can depend on the available storage space on or off the chip, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in the inference and / or training of the neural network, or some combination of these factors.

[0088] In at least one embodiment, the inference and / or training logic 715 can include, but is not limited to, code and / or data storage 705 to store the backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained as and / or used for inference in aspects of one or more embodiments. In at least one embodiment, during training and / or inference using aspects of one or more embodiments, the code and / or data storage 705 stores the weight parameters and / or input / output data of each layer of the neural network used or trained in conjunction with one or more embodiments during the backpropagation of the input / output data and / or weight parameters. In at least one embodiment, the training logic 715 can include or be coupled to code and / or data storage 705 for storing graph code or other software to control timing and / or sequencing, where the weights and / or other parameter information are loaded to configure the logic, which includes integer and / or floating-point units (collectively referred to as arithmetic logic units (ALUs)).

[0089] In at least one embodiment, the code (such as graph code) causes the weights or other parameter information to be loaded into the processor ALU based on the architecture of the neural network corresponding to the code. In at least one embodiment, any portion of the code and / or data storage 705 can be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of the code and / or data storage 705 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 705 can be cache memory, DRAM, SRAM, non-volatile memory (such as flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage 705 is internal or external to the processor, e.g., whether it consists of DRAM, SRAM, flash memory, or some other storage type, depends on whether the available storage is on-chip or off-chip, the latency requirements of the training and / or inference functions being performed, the data batch size used in the inference and / or training of the neural network, or some combination of these factors.

[0090] In at least one embodiment, code and / or data store 701 and code and / or data store 705 may be separate storage structures. In at least one embodiment, code and / or data store 701 and code and / or data store 705 may be combined storage structures. In at least one embodiment, code and / or data store 701 and code and / or data store 705 may be partially combined and partially separated. In at least one embodiment, any portion of code and / or data store 701 and code and / or data store 705 may be included with other on-chip or off-chip data stores, including an L1, L2, or L3 cache of the processor or system memory.

[0091] In at least one embodiment, inference and / or training logic 715 may include, but is not limited to, one or more arithmetic logic units (“ALUs”) 710 (including integer and / or floating point units) for performing logical and / or mathematical operations at least in part based on training and / or inference code (e.g., graph code) or as directed thereby, the results of which may produce activations (e.g., output values from layers or neurons internal to a neural network) stored in activation store 720, which are a function of input / output and / or weight parameter data stored in code and / or data store 701 and / or code and / or data store 705. In at least one embodiment, the activations are generated by performing linear algebra and / or matrix-based mathematics by ALU 710 in response to executing instructions or other code, where weight values stored in code and / or data store 705 and / or data store 701 are used as operands with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in code and / or data store 705 or code and / or data store 701 or another on-chip or off-chip store.

[0092] In at least one embodiment, one or more ALUs 710 are included in one or more processors or other hardware logic devices or circuits, while in another embodiment, one or more ALUs 710 may be outside the processor or other hardware logic devices or circuits that use them (such as coprocessors). In at least one embodiment, one or more ALUs 710 may be included within the execution unit of a processor or otherwise included in a group of ALUs accessible by the execution unit of a processor, and the execution unit of the processor may be within the same processor or distributed among different processors of different types (e.g., central processing unit, graphics processing unit, fixed function unit, etc.). In at least one embodiment, code and / or data storage 701, code and / or data storage 705, and activation storage 720 may share a processor or other hardware logic device or circuit, while in another embodiment, they may be in different processors or other hardware logic devices or circuits or some combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation storage 720 may be included with other on-chip or off-chip data storage, including the L1, L2, or L3 cache of the processor or system memory. Additionally, the inference and / or training code may be stored together with other code accessible by the processor or other hardware logic or circuits and may be extracted and / or processed using the fetch, decode, schedule, execute, retire, and / or other logic circuits of the processor.

[0093] In at least one embodiment, activation storage 720 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, activation storage 720 may be wholly or partially inside or outside one or more processors or other logic circuits. In at least one embodiment, depending on the on-chip or off-chip available storage, the latency requirements of the training and / or inference functions, the batch size of the data used in the inference and / or training neural network, or some combination of these factors, activation storage 720 may be selected to be internal or external to the processor, e.g., or include DRAM, SRAM, flash memory, or some other storage type.

[0094] In at least one embodiment, Figure 7A the inference and / or training logic 715 shown may be used in conjunction with an application specific integrated circuit (“ASIC”), such as the processing unit from Google, the TM inference processing unit (IPU) from Graphcore, or the (e.g., “Lake Crest”) processor from Intel Corp. In at least one embodiment, Figure 7AThe inference and / or training logic 715 shown can be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware, hardware, or other hardware (such as field programmable gate array (“FPGA”)).

[0095] Figure 7B An inference and / or training logic 715 according to at least one embodiment is shown. In at least one embodiment, the inference and / or training logic 715 can include, but is not limited to, hardware logic where computing resources are dedicated or otherwise uniquely used along with weight values or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, Figure 7B the inference and / or training logic 715 shown can be used in conjunction with an application specific integrated circuit (ASIC), such as the processing unit from Google, the TM inference processing unit (IPU) from Graphcore or the (e.g., “Lake Crest”) processor from Intel Corp. In at least one embodiment, Figure 7B the inference and / or training logic 715 shown can be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, hardware, or other hardware (such as field programmable gate array (FPGA)). In at least one embodiment, the inference and / or training logic 715 includes, but is not limited to, code and / or data storage 701 and code and / or data storage 705, which can be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. In Figure 7B at least one embodiment shown, each of code and / or data storage 701 and code and / or data storage 705 is respectively associated with dedicated computing resources (such as computing hardware 702 and computing hardware 706). In at least one embodiment, each of computing hardware 702 and computing hardware 706 includes one or more ALUs that only perform mathematical functions (such as linear algebra functions) on the information stored in code and / or data storage 701 and code and / or data storage 705 respectively, and the result of the executed function is stored in activation storage 720.

[0096] In at least one embodiment, each of code and / or data stores 701 and 705 and corresponding computing hardware 702 and 706 respectively corresponds to a different layer of a neural network such that activations obtained from one storage / compute pair 701 / 702 of code and / or data store 701 and computing hardware 702 are provided as input to the next storage / compute pair 705 / 706 of code and / or data store 705 and computing hardware 706 in order to reflect the conceptual organization of the neural network. In at least one embodiment, each storage / compute pair 701 / 702 and 705 / 706 can correspond to more than one neural network layer. In at least one embodiment, additional storage / compute pairs (not shown) can be included in inference and / or training logic 715 after or in parallel with storage / compute pairs 701 / 702 and 705 / 706.

[0097] Neural Network Training and Deployment

[0098] Figure 8 Illustrated is the training and deployment of a deep neural network according to at least one embodiment. In at least one embodiment, an untrained neural network 806 is trained using a training data set 802. In at least one embodiment, the training framework 804 is the PyTorch framework, while in other embodiments, the training framework 804 is TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j or other training frameworks. In at least one embodiment, the training framework 804 trains the untrained neural network 806 and enables it to be trained using the processing resources described herein to generate a trained neural network 808. In at least one embodiment, the weights can be randomly selected or pre-trained using a deep belief network. In at least one embodiment, training can be performed in a supervised, partially supervised or unsupervised manner.

[0099] In at least one embodiment, supervised learning is used to train an untrained neural network 806, where the training dataset 802 includes inputs paired with desired outputs for the inputs, or where the training dataset 802 includes inputs with known outputs and the outputs of the neural network 806 are manually graded. In at least one embodiment, the untrained neural network 806 is trained in a supervised manner, and the inputs from the training dataset 802 are processed and the resulting outputs are compared with a set of desired or wanted outputs. In at least one embodiment, the error is then propagated back through the untrained neural network 806. In at least one embodiment, the training framework 804 adjusts the weights that control the untrained neural network 806. In at least one embodiment, the training framework 804 includes tools for monitoring the degree to which the untrained neural network 806 converges to a model (e.g., a trained neural network 808), a model adapted to generate correct answers (e.g., results 814) based on input data (e.g., a new dataset 812). In at least one embodiment, the training framework 804 repeatedly trains the untrained neural network 806 while adjusting the weights to improve the output of the untrained neural network 806 using a loss function and an adjustment algorithm (e.g., stochastic gradient descent). In at least one embodiment, the training framework 804 trains the untrained neural network 806 until the untrained neural network 806 reaches a desired accuracy. In at least one embodiment, the trained neural network 808 can then be deployed to perform any number of machine learning operations.

[0100] In at least one embodiment, unsupervised learning is used to train an untrained neural network 806, where the untrained neural network 806 attempts to train itself using unlabeled data. In at least one embodiment, the unsupervised learning training dataset 802 will include input data without any associated output data or "ground truth" data. In at least one embodiment, the untrained neural network 806 can learn groupings within the training dataset 802 and can determine how individual inputs relate to the untrained dataset 802. In at least one embodiment, unsupervised training can be used to generate a self-organizing map in the trained neural network 808, which can perform operations useful for reducing the dimensionality of a new dataset 812. In at least one embodiment, unsupervised training can also be used to perform anomaly detection, which allows identification of data points in a new dataset 812 that deviate from the normal pattern of the new dataset 812.

[0101] In at least one embodiment, semi-supervised learning may be used, which is a technique in which a mixture of labeled data and unlabeled data is included in a training data set 802. In at least one embodiment, a training framework 804 may be used to perform incremental learning, for example, by transfer learning techniques. In at least one embodiment, incremental learning enables a trained neural network 808 to adapt to a new data set 812 without forgetting the knowledge injected into the trained neural network 808 during initial training.

[0102] Referring Figure 9 , Figure 9 FIG. is an example data flow diagram of a process 900 for generating and deploying a processing and inference pipeline according to at least one embodiment. In at least one embodiment, the process 900 may be deployed to perform game name recognition analysis and inference on user feedback data at one or more facilities 902 such as a data center.

[0103] In at least one embodiment, the process 900 may be executed within a training system 904 and / or a deployment system 906. In at least one embodiment, the training system 904 may be used to perform training, deployment, and implementation of machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for the deployment system 906. In at least one embodiment, the deployment system 906 may be configured to offload processing and computing resources in a distributed computing environment to reduce the infrastructure requirements of the facility 902. In at least one embodiment, the deployment system 906 may provide a pipeline platform for selecting, customizing, and implementing virtual instruments for use with computing devices at the facility 902. In at least one embodiment, the virtual instrument may include a software-defined application for performing one or more processing operations on feedback data. In at least one embodiment, one or more applications in the pipeline may use or invoke services (e.g., inference, visualization, computing, AI, etc.) of the deployment system 906 during application execution.

[0104] In at least one embodiment, some applications used in an advanced processing and inference pipeline may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, feedback data 908 (e.g., feedback data) stored at the facility 902 or feedback data 908 from another or more facilities may be used at the facility 902, or a combination thereof, to train a machine learning model. In at least one embodiment, the training system 904 may be used to provide applications, services, and / or other resources to generate a working, deployable machine learning model for the deployment system 906.

[0105] In at least one embodiment, the model registry 924 can be supported by an object store, which can support version control and object metadata. In at least one embodiment, the object store can be accessed from within the cloud platform via, for example, an application programming interface (API) compatible with cloud storage (e.g., Figure 10 cloud 1026). In at least one embodiment, machine learning models within the model registry 924 can be uploaded, listed, modified, or deleted by developers or partners of systems that interact with the API. In at least one embodiment, the API can provide access to methods that allow users with appropriate credentials to associate a model with an application such that the model can be executed as part of the execution of a containerized instantiation of the application.

[0106] In at least one embodiment, the training pipeline 1004 ( Figure 10 ) can include scenarios where the facility 902 is training their own machine learning models or has existing machine learning models that need to be optimized or updated. In at least one embodiment, feedback data 908 can be received from various channels such as forums, web forms, etc. In at least one embodiment, once the feedback data 908 is received, AI-assisted annotation 910 can be used to help generate annotations corresponding to the feedback data 908 to be used as ground truth data for the machine learning model. In at least one embodiment, the AI-assisted annotation 910 can include one or more machine learning models (e.g., a convolutional neural network (CNN)) that can be trained to generate annotations corresponding to certain types of feedback data 908 (e.g., from certain devices) and / or certain types of anomalies in the feedback data 908. In at least one embodiment, the AI-assisted annotation 910 can then be used directly or can be adjusted or fine-tuned using annotation tools to generate the ground truth data. In at least one embodiment, in some examples, the labeled data 912 can be used as the ground truth data for training the machine learning model. In at least one embodiment, the AI-assisted annotation 910, the labeled data 912, or a combination thereof can be used as the ground truth data for training the machine learning model. In at least one embodiment, the trained machine learning model can be referred to as the output model 916 and can be used by the deployment system 906 as described herein.

[0107] In at least one embodiment, the training pipeline 1004 ( Figure 10) may include the following scenarios: where the facility 902 requires a machine learning model to perform one or more processing tasks for deploying one or more applications in the deployment system 906, but the facility 902 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for this purpose). In at least one embodiment, an existing machine learning model can be selected from the model registry 924. In at least one embodiment, the model registry 924 may include machine learning models that are trained to perform various different inference tasks on imaging data. In at least one embodiment, the machine learning models in the model registry 924 can be trained on imaging data from different facilities (e.g., a facility located remotely) rather than the facility 902. In at least one embodiment, the machine learning model may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when trained on imaging data from a specific location, the training can be performed at that location or at least in a manner that protects the confidentiality of the imaging data or restricts the off-site transfer of the imaging data (e.g., in compliance with HIPAA regulations, privacy regulations, etc.). In at least one embodiment, once the model or a portion of the model has been trained at one location, the machine learning model can be added to the model registry 924. In at least one embodiment, the machine learning model can then be retrained or updated at any number of other facilities, and the retrained or updated model can be used in the model registry 924. In at least one embodiment, a machine learning model (referred to as the output model 916) can then be selected from the model registry 924 and used in the deployment system 906 to perform one or more processing tasks for one or more applications of the deployment system.

[0108] In at least one embodiment, the training pipeline 1004( Figure 10)Can be used in scenarios including facility 902, which requires a machine learning model for performing one or more processing tasks for deploying one or more applications in system 906, but facility 902 may not currently have such a machine learning model (or may not have an optimized, efficient, or effective model for this purpose). In at least one embodiment, due to population differences, genetic variations, robustness of the training data for training the machine learning model, diversity of training data anomalies, and / or other issues with the training data, the machine learning model selected from model registry 924 may not be fine-tuned or optimized for the feedback data 908 generated at facility 902. In at least one embodiment, AI-assisted annotation 910 can be used to help generate annotations corresponding to feedback data 908 for use as ground truth data for retraining or updating the machine learning model. In at least one embodiment, labeled data 912 can be used as ground truth data for training the machine learning model. In at least one embodiment, retraining or updating the machine learning model can be referred to as model training 914. In at least one embodiment, model training 914 (e.g., AI-assisted annotation 910, labeled data 912, or a combination thereof) can be used as ground truth data for retraining or updating the machine learning model.

[0109] In at least one embodiment, deployment system 906 can include software 918, services 920, hardware 922, and / or other components, features, and functions. In at least one embodiment, deployment system 906 can include a software "stack" such that software 918 can be built on top of services 920 and services 920 can be used to perform some or all of the processing tasks, and services 920 and software 918 can be built on top of hardware 922 and hardware 922 can be used to perform the processing, storage, and / or other computing tasks of deployment system 906.

[0110] In at least one embodiment, software 918 may include any number of different containers, where each container may execute an instantiation of an application. In at least one embodiment, each application may perform one or more processing tasks (e.g., inference, object detection, feature detection, segmentation, image enhancement, calibration, etc.) in an advanced processing and inference pipeline. In at least one embodiment, for each type of computing device, there may be any number of containers that may perform data processing tasks on feedback data 908 (or other data types, such as the data types described herein). In at least one embodiment, in addition to the containers that receive and configure imaging data for use by each container and / or for use by facility 902 after processing through the pipeline, an advanced processing and inference pipeline may be defined based on the selection of different containers desired or required for processing feedback data 908 (e.g., to convert the output back to a usable data type for storage and display at facility 902). In at least one embodiment, a combination of containers within software 918 (e.g., which constitutes a pipeline) may be referred to as a virtual instrument (as described in more detail herein), and the virtual instrument may utilize services 920 and hardware 922 to perform some or all of the processing tasks of the applications instantiated in the containers.

[0111] In at least one embodiment, data may be preprocessed as part of a data processing pipeline to prepare the data for processing by one or more applications. In at least one embodiment, post-processing may be performed on the output of one or more inference tasks or other processing tasks in the pipeline to prepare the output data for the next application and / or to prepare the output data for transmission and / or use by a user (e.g., as a response to an inference request). In at least one embodiment, the inference tasks may be performed by one or more machine learning models, such as a trained or deployed neural network, which may include output model 916 of training system 904.

[0112] In at least one embodiment, the tasks of a data processing pipeline may be encapsulated in one or more containers, where each container represents a discrete, fully functional instantiation of an application and a virtualized computing environment that is capable of referencing a machine learning model. In at least one embodiment, a container or application may be published to a private (e.g., limited access) area of a container registry (described in more detail herein), and a trained or deployed model may be stored in model registry 924 and associated with one or more applications. In at least one embodiment, an image of an application (e.g., a container image) may be used in the container registry, and once a user selects an image from the container registry for deployment in the pipeline, the image may be used to generate a container for instantiation of the application for use by the user's system.

[0113] In at least one embodiment, a developer can develop, publish, and store an application (e.g., as a container) for performing processing and / or inference on the provided data. In at least one embodiment, a software development kit (SDK) associated with the system can be used to perform development, publishing, and / or storage (e.g., to ensure that the developed application and / or container conforms to or is compatible with the system). In at least one embodiment, the developed application can be tested locally using the SDK (e.g., at a first facility, on data from the first facility), and the SDK, as the system (e.g., Figure 10 system 1000 in) can support at least some services 920. In at least one embodiment, once verified by the system 1000 (e.g., for accuracy, etc.), the application becomes available in the container registry for users (e.g., hospitals, clinics, laboratories, healthcare providers, etc.) to select and / or implement to perform one or more processing tasks on data at the users' facilities (e.g., a second facility).

[0114] In at least one embodiment, the developer can then share the application or container over a network for access and use by users of the system (e.g., Figure 10 system 1000). In at least one embodiment, the completed and verified application or container can be stored in the container registry, and the associated machine learning model can be stored in the model registry 924. In at least one embodiment, a requesting entity (which provides an inference or image processing request) can browse the container registry and / or the model registry 924 for applications, containers, data sets, machine learning models, etc., select the desired combination of elements to include in a data processing pipeline, and submit a processing request. In at least one embodiment, the request can include the input data necessary to execute the request, and / or can include the selection of an application and / or machine learning model to be executed when processing the request. In at least one embodiment, the request can then be passed to one or more components (e.g., the cloud) of the deployment system 906 to perform the processing of the data processing pipeline. In at least one embodiment, the processing performed by the deployment system 906 can include referencing the elements selected from the container registry and / or the model registry 924 (e.g., applications, containers, models, etc.). In at least one embodiment, once the results are generated by the pipeline, the results can be returned to the user for reference (e.g., for viewing in a viewing application suite executed locally, on a local workstation, or terminal).

[0115] In at least one embodiment, to assist in processing or executing applications or containers in a pipeline, service 920 may be utilized. In at least one embodiment, service 920 may include a computing service, an artificial intelligence (AI) service, a visualization service, and / or other service types. In at least one embodiment, service 920 may provide functions common to one or more applications in software 918, and thus functions may be abstracted as services that can be invoked or utilized by applications. In at least one embodiment, the functions provided by service 920 may run dynamically and more efficiently, while also scaling well by allowing applications to process data in parallel (e.g., using Figure 10 the parallel computing platform 1030 therein). In at least one embodiment, rather than requiring each application that shares the same function provided by service 920 to have a corresponding instance of service 920, service 920 may be shared among and within various applications. In at least one embodiment, by way of non-limiting example, the service may include an inference server or engine that can be used to perform detection or segmentation tasks. In at least one embodiment, a model training service may be included, which may provide machine learning model training and / or retraining capabilities.

[0116] In at least one embodiment, in the case where service 920 includes an AI service (e.g., an inference service), as part of application execution, by invoking (e.g., as an API call) the inference service (e.g., an inference server), to perform one or more machine learning models or their processing, one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth anomalies, scar formation, etc.) may be executed. In at least one embodiment, in the case where another application includes one or more machine learning models for a segmentation task, the application may call the inference service to execute the machine learning model for performing one or more processing operations associated with the segmentation task. In at least one embodiment, the software 918 implementing the advanced processing and inference pipeline may be pipelined because each application may call the same inference service to perform one or more inference tasks.

[0117] In at least one embodiment, the hardware 922 may include a GPU, a CPU, a graphics card, an AI / deep learning system (e.g., an AI supercomputer such as NVIDIA's DGX supercomputer system), a cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 922 may be used to provide efficient, specially built support for the software 918 and services 920 in the deployment system 906. In at least one embodiment, GPU processing may be implemented for local processing (e.g., at the facility 902) within an AI / deep learning system, in a cloud system, and / or in other processing components of the deployment system 906 to improve the efficiency, accuracy, and performance of game name recognition.

[0118] In at least one embodiment, by way of non-limiting example, with respect to deep learning, machine learning, and / or high-performance computing, the software 918 and / or services 920 may be optimized for GPU processing. In at least one embodiment, at least some of the computing environments of the deployment system 906 and / or the training system 904 may be executed in a data center, one or more supercomputers, or high-performance computer systems having GPU-optimized software (e.g., the hardware and software combination of an NVIDIA DGX system). In at least one embodiment, as described herein, the hardware 922 may include any number of GPUs that may be invoked to perform data processing in parallel. In at least one embodiment, the cloud platform may also include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computing tasks. In at least one embodiment, an AI / deep learning supercomputer and / or GPU-optimized software (e.g., as provided on NVIDIA's DGX systems) may be used as a hardware abstraction and scaling platform to execute a cloud platform (e.g., NVIDIA's NGC). In at least one embodiment, the cloud platform may integrate an application container cluster system or a coordination system (e.g., KUBERNETES) across multiple GPUs to enable seamless scaling and load balancing.

[0119] Figure 10 is a system diagram of an example system 1000 for generating and deploying a deployment pipeline according to at least one embodiment. In at least one embodiment, the system 1000 may be used to implement Figure 9 process 900 and / or other processes, including advanced processing and inference pipelines. In at least one embodiment, the system 1000 may include a training system 904 and a deployment system 906. In at least one embodiment, the training system 904 and the deployment system 906 may be implemented using the software 918, services 920, and / or hardware 922 as described herein.

[0120] In at least one embodiment, system 1000 (e.g., training system 904 and / or deployment system 906) may be implemented in a cloud computing environment (e.g., using cloud 1026). In at least one embodiment, system 1000 may be implemented locally (with respect to a facility), or as a combination of cloud computing resources and local computing resources. In at least one embodiment, access to APIs in cloud 1026 may be restricted to authorized users by establishing security measures or protocols. In at least one embodiment, the security protocol may include a network token, which may be signed by an authentication (e.g., AuthN, AuthZ, Gluecon, etc.) service and may carry appropriate authorization. In at least one embodiment, the APIs of virtual instruments (described herein) or other instances of system 1000 may be restricted to a set of public IPs that have been audited or authorized for interaction.

[0121] In at least one embodiment, the various components of system 1000 may communicate with each other and among themselves using any of a variety of different network types, including but not limited to local area networks (LANs) and / or wide area networks (WANs) via wired and / or wireless communication protocols. In at least one embodiment, communication between the facilities and components of system 1000 (e.g., for sending inference requests, for receiving results of inference requests, etc.) may be conveyed via one or more data buses, wireless data protocols (Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0122] In at least one embodiment, similar to that described herein with respect to Figure 9 As described, training system 904 may execute training pipeline 1004. In at least one embodiment, where deployment system 906 will use one or more machine learning models in deployment pipeline data 1660, training pipeline 1004 may be used to train or retrain one or more (e.g., pre-trained) models, and / or implement one or more pre-trained models 1006 (e.g., without retraining or updating). In at least one embodiment, as a result of training pipeline 1004, output model 916 may be generated. In at least one embodiment, training pipeline 1004 may include any number of processing steps, AI-assisted annotation 910, tagging or annotation of feedback data 908 to generate tagged data 912, selection of a model from a model registry, model training 914, training, retraining, or updating of a model, and / or other processing steps. In at least one embodiment, different training pipelines 1004 may be used for different machine learning models used by deployment system 906. In at least one embodiment, similar to training pipeline 1004 of the first example described with respect to Figure 9 described may be used for a first machine learning model, similar to that described with respect to Figure 9The training pipeline 1004 of the second example described can be used for a second machine learning model, similar to the Figure 9 The training pipeline 1004 of the third example described can be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 904 can be used according to the requirements of each respective machine learning model. In at least one embodiment, one or more machine learning models may already be trained and ready for deployment, so the training system 904 may not perform any processing on the machine learning models, and the machine learning models can be implemented by the deployment system 906.

[0123] In at least one embodiment, according to the embodiment or embodiments, one or more output models 916 and / or pre-trained models 1006 can include any type of machine learning model. In at least one embodiment and without limitation, the machine learning models used by the system 1000 can include those using linear regression, logistic regression, decision trees, support vector machines (SVMs), naive Bayes, k-nearest neighbors (Knn), k-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutional, recurrent, perceptron, long / short-term memory (LSTM), Bi-LSTM, Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid state machines, etc.), and / or other types of machine learning models.

[0124] In at least one embodiment, the training pipeline 1004 can include AI-assisted annotation. In at least one embodiment, labeled data 912 (e.g., traditional annotation) can be generated by any number of techniques. In at least one embodiment, in some examples, labels or other annotations can be generated in a drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a markup program, another type of application suitable for generating ground truth annotations or labels, and / or can be hand-drawn. In at least one embodiment, ground truth data can be synthetically generated (e.g., generated from a computer model or rendering), real-world generated (e.g., designed and generated from real-world data), machine-generated automatically (e.g., using feature analysis and learning to extract features from data and then generate labels), manually annotated (e.g., by a tagger or annotation expert to define the location of the label), and / or combinations thereof. In at least one embodiment, for each instance of feedback data 908 (or other data types used by the machine learning model), there can be corresponding ground truth data generated by the training system 904. In at least one embodiment, AI-assisted annotation can be performed as part of the deployment pipeline data 1660; supplementing or replacing the AI-assisted annotation included in the training pipeline 1004. In at least one embodiment, the system 1000 can include a multi-layer platform, and the multi-layer platform can include a software layer (e.g., software 918) of a diagnostic application (or other application type), which can perform one or more medical imaging and diagnostic functions.

[0125] In at least one embodiment, the software layer can be implemented as a secure, encrypted, and / or certified API through which an application or container can be invoked (e.g., called) from an external environment (e.g., facility 902). In at least one embodiment, the application can then call or execute one or more services 920 to perform computational, AI, or visualization tasks associated with the respective application, and the software 918 and / or the services 920 can utilize the hardware 922 to perform processing tasks in an effective and efficient manner.

[0126] In at least one embodiment, the deployment system 906 can execute the deployment pipeline data 1660. In at least one embodiment, the deployment pipeline data 1660 can include any number of applications, which can be sequential, non-sequential, or otherwise applied to the feedback data (and / or other data types) - including AI-assisted annotation, as described above. In at least one embodiment, as described herein, the deployment pipeline data 1660 for an individual device can be referred to as a virtual instrument for the device. In at least one embodiment, for a single device, there can be more than one deployment pipeline data 1660, depending on the information desired from the data generated by the device.

[0127] In at least one embodiment, the applications that can be used to deploy pipeline data 1660 can include any application that can be used to perform processing tasks on feedback data or other data from a device. In at least one embodiment, since various applications can share common image operations, in some embodiments, a data augmentation library (e.g., as one of the services 920) can be used to accelerate these operations. In at least one embodiment, to avoid the bottleneck of traditional processing methods that rely on CPU processing, the parallel computing platform 1030 can be used for GPU acceleration of these processing tasks.

[0128] In at least one embodiment, the deployment system 906 can include user interface data 1664 (e.g., a graphical user interface, a web interface, etc.), and the user interface can be used to select the applications to be included in the deployment pipeline data 1660, arrange the applications, modify or change the applications or their parameters or configurations, use and interact with the deployment pipeline data 1660 during setup and / or deployment, and / or otherwise interact with the deployment system 906. In at least one embodiment, although not shown with respect to the training system 904, the user interface data 1664 (or a different user interface) can be used to select the models to be used in the deployment system 906, to select the models for training or retraining in the training system 904, and / or to otherwise interact with the training system 904.

[0129] In at least one embodiment, in addition to the application coordination system 1028, pipeline manager data 1662 can also be used to manage the interaction between the applications or containers of the deployment pipeline data 1660 and the services 920 and / or the hardware 922. In at least one embodiment, the pipeline manager data 1662 can be configured to facilitate interactions from application to application, from application to service 920, and / or from application or service to hardware 922. In at least one embodiment, although shown as being included in the software 918, this is not intended to be limiting, and in some examples, the pipeline manager data 1662 can be included in the service 920. In at least one embodiment, the application coordination system 1028 (e.g., Kubernetes, DOCKER, etc.) can include a container coordination system that can group applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by associating the applications from the deployment pipeline data 1660 (e.g., reconstruction applications, segmentation applications, etc.) with individual containers, each application can execute in a self - contained environment (e.g., at the kernel level) to improve speed and efficiency.

[0130] In at least one embodiment, each application and / or container (or its image) can be developed, modified, and deployed separately (e.g., a first user or developer can develop, modify, and deploy a first application, and a second user or developer can develop, modify, and deploy a second application separate from the first user or developer), which can allow for focusing on and attending to the tasks of a single application and / or container without being hindered by the tasks of another application or container. In at least one embodiment, the pipeline manager data 1662 and the application coordination system 1028 can assist in the communication and collaboration between different containers or applications. In at least one embodiment, as long as the expected inputs and / or outputs of each container or application are known to the system (e.g., based on the construction of the application or container), the application coordination system 1028 and / or the pipeline manager data 1662 can facilitate communication and resource sharing between and among each application or container. In at least one embodiment, since one or more applications or containers in the deployment pipeline data 1660 can share the same services and resources, the application coordination system 1028 can coordinate, load balance, and determine the sharing of services or resources between and among the various applications or containers. In at least one embodiment, a scheduler can be used to track the resource requirements of an application or container, the current or planned usage of those resources, and the resource availability. Thus, in at least one embodiment, the scheduler can allocate resources to different applications, taking into account the requirements and availability of the system, and distribute resources between and among the applications. In some examples, the scheduler (and / or other components of the application coordination system 1028) can determine resource availability and distribution based on constraints imposed on the system (e.g., user constraints), such as quality of service (QoS), the urgency of data output (e.g., to determine whether to perform real-time processing or deferred processing), etc.

[0131] In at least one embodiment, services 920 utilized and shared by an application or container in a deployment system 906 may include compute service data 1666, AI service data 1668, visualization service 1020, and / or other service types. In at least one embodiment, an application may invoke (e.g., execute) one or more services 920 to perform processing operations for the application. In at least one embodiment, an application may utilize compute service data 1666 to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, one or more compute service data 1666 may be utilized to perform parallel processing (e.g., using parallel computing platform 1030) to process data substantially simultaneously by one or more applications and / or one or more tasks of a single application. In at least one embodiment, parallel computing platform 1030 (e.g., NVIDIA's CUDA) may implement general-purpose computing on a GPU (GPGPU) (e.g., GPU 1022). In at least one embodiment, the software layer of parallel computing platform 1030 may provide access to the virtual instruction set and parallel computing elements of the GPU to execute compute kernels. In at least one embodiment, parallel computing platform 1030 may include memory, and in some embodiments, the memory may be shared among and within multiple containers and / or among and within different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and / or multiple processes within a container to use the same data from a shared memory segment of parallel computing platform 1030 (e.g., where multiple different stages of one application or multiple applications are processing the same information). In at least one embodiment, rather than copying data and moving it to different locations in memory (e.g., read / write operations), the same data in the same location in memory may be used for any number of processing tasks (e.g., at the same time, different times, etc.). In at least one embodiment, since data is used as a result of processing to generate new data, this information about the new location of the data may be stored and shared among various applications. In at least one embodiment, the location of the data and the location of the updated or modified data may be part of the definition of how to understand the payload in a container.

[0132] In at least one embodiment, AI service data 1668 can be utilized to perform an inference service for executing a machine learning model associated with an application (e.g., the task is to perform one or more processing tasks of the application). In at least one embodiment, the AI service data 1668 can utilize the AI system 1024 to execute a machine learning model (e.g., a neural network such as a CNN) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, the application of the deployment pipeline data 1660 can use one or more output models 916 from the training system 904 and / or other models of the application to perform inference on imaging data (e.g., DICOM data, RIS data, CIS data, REST-compliant data, RPC data, raw data, etc.). In at least one embodiment, two or more examples of performing inference using the application coordination system 1028 (e.g., a scheduler) can be available. In at least one embodiment, the first category can include a high-priority / low-latency path, which can implement a higher service level agreement, such as for performing inference on emergency requests in an emergency situation or for a radiologist during a diagnostic process. In at least one embodiment, the second category can include a standard-priority path, which can be used for requests that may not be urgent or for situations where analysis can be performed at a later time. In at least one embodiment, the application coordination system 1028 can allocate resources (e.g., service 920 and / or hardware 922) based on the priority path for different inference tasks of the AI service data 1668.

[0133] In at least one embodiment, a shared memory may be installed to the AI service data 1668 in the system 1000. In at least one embodiment, the shared memory may operate as a cache (or other storage device type), and may be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, a set of API instances of the deployment system 906 may receive the request, and may select one or more instances (e.g., for best fit, for load balancing, etc.) to process the request. In at least one embodiment, to process the request, the request may be input into a database, and if not already in the cache, the machine learning model may be located from the model registry 924, the validation step may ensure that the appropriate machine learning model is loaded into the cache (e.g., shared storage), and / or a copy of the model may be saved to the cache. In at least one embodiment, if the application is not already running or there are not enough instances of the application, a scheduler (e.g., the scheduler of the pipeline manager data 1662) may be used to start the application referenced in the request. In at least one embodiment, if the inference server has not been started to execute the model, the inference server may be started. In at least one embodiment, each model may start any number of inference servers. In at least one embodiment, in a pull model where the inference servers are clustered, the model may be cached whenever load balancing is beneficial. In at least one embodiment, the inference servers may be statically loaded into the corresponding distributed servers.

[0134] In at least one embodiment, an inference server running in a container may be used to perform inference. In at least one embodiment, an instance of the inference server may be associated with a model (and optionally with multiple versions of the model). In at least one embodiment, if an instance of the inference server does not exist when a request to perform inference on a model is received, a new instance may be loaded. In at least one embodiment, when the inference server is started, the model may be passed to the inference server such that the same container may be used to serve different models as long as the inference server runs as different instances.

[0135] In at least one embodiment, during application execution, an inference request for a given application can be received, and a container (e.g., an instance of a hosted inference server) can be loaded (if not already loaded), and a launcher can be invoked. In at least one embodiment, preprocessing logic in the container can load, decode, and / or perform any additional preprocessing on the incoming data (e.g., using a CPU and / or GPU). In at least one embodiment, once the data is ready for inference, the container can perform inference on the data as needed. In at least one embodiment, this can include a single inference call on an image (e.g., a hand X-ray), or may require inference on hundreds of images (e.g., chest CTs). In at least one embodiment, the application can summarize the results before completion, which can include but is not limited to a single confidence score, pixel-level segmentation, voxel-level segmentation, generating visualizations, or generating text to summarize the results. In at least one embodiment, different priorities can be assigned to different models or applications. For example, some models can have real-time (TAT less than 1 minute) priority, while other models can have a lower priority (e.g., TAT less than 10 minutes). In at least one embodiment, the model execution time can be measured from the requesting agency or entity and can include the cooperative network traversal time as well as the execution time of the inference service.

[0136] In at least one embodiment, the transfer of requests between the service 920 and the inference application can be hidden behind a software development kit (SDK) and can provide a robust transfer via a queue. In at least one embodiment, requests will be placed in the queue via an API for an individual application / tenant ID combination, and the SDK will pull requests from the queue and provide the requests to the application. In at least one embodiment, the name of the queue can be provided in the environment from which the SDK will pick up the queue. In at least one embodiment, asynchronous communication via the queue can be useful as it can allow any instance of the application to pick up work when it is available. In at least one embodiment, the results can be transferred back via the queue to ensure no data is lost. In at least one embodiment, the queue can also provide the ability to split the work, as the highest priority work can go into the queue connected to most instances of the application, while the lowest priority work can go into the queue connected to a single instance that processes tasks in the order received. In at least one embodiment, the application can run on a GPU-accelerated instance generated in the cloud 1026, and the inference service can perform inference on the GPU.

[0137] In at least one embodiment, a visualization service 1020 can be utilized to generate visualizations for viewing the output of application and / or deployment pipeline data 1660. In at least one embodiment, the visualization service 1020 can utilize a GPU 1022 to generate visualizations. In at least one embodiment, the visualization service 1020 can implement rendering effects such as ray tracing to generate higher quality visualizations. In at least one embodiment, visualizations can include, but are not limited to, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomographic slices, virtual reality displays, augmented reality displays, etc. In at least one embodiment, a virtualized environment can be used to generate a virtual interactive display or environment (e.g., a virtual environment) for interaction by system users (e.g., doctors, nurses, radiologists, etc.). In at least one embodiment, the visualization service 1020 can include an internal visualizer, movie, and / or other rendering or image processing capabilities or functions (e.g., ray tracing, rasterization, internal optics, etc.).

[0138] In at least one embodiment, the hardware 922 can include a GPU 1022, an AI system 1024, a cloud 1026, and / or any other hardware for executing the training system 904 and / or the deployment system 906. In at least one embodiment, the GPU 1022 (e.g., NVIDIA's TESLA and / or QUADRO GPUs) can include any number of GPUs capable of processing tasks for any features or functions that can be used to execute the computing service data 1666, the AI service data 1668, the visualization service 1020, other services, and / or software 918. For example, for the AI service data 1668, the GPU 1022 can be used to perform preprocessing on imaging data (or other data types used by machine learning models), perform postprocessing on the output of machine learning models, and / or perform inference (e.g., to execute a machine learning model). In at least one embodiment, the cloud 1026, the AI system 1024, and / or other components of the system 1000 can use the GPU 1022. In at least one embodiment, the cloud 1026 can include a GPU-optimized platform for deep learning tasks. In at least one embodiment, the AI system 1024 can use GPUs, and one or more AI systems 1024 can be used to execute the cloud 1026 (or at least part of the tasks for deep learning or inference). Similarly, although the hardware 922 is shown as discrete components, this is not intended to be limiting, and any component of the hardware 922 can be combined with or utilized by any other component of the hardware 922.

[0139] In at least one embodiment, the AI system 1024 may include a specially constructed computing system (e.g., a supercomputer or HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, in addition to the CPU, RAM, memory, and / or other components, features, or functions, the AI system 1024 (e.g., NVIDIA's DGX) may also include software (e.g., a software stack) that can use multiple GPUs 1022 to perform sub-GPU optimization. In at least one embodiment, one or more AI systems 1024 may be implemented in the cloud 1026 (e.g., in a data center) to perform some or all of the AI-based processing tasks of the system 1000.

[0140] In at least one embodiment, the cloud 1026 may include GPU-accelerated infrastructure (e.g., NVIDIA's NGC), which may provide a GPU-optimized platform for performing the processing tasks of the system 1000. In at least one embodiment, the cloud 1026 may include an AI system 1024 for performing one or more AI-based tasks of the system 1000 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, the cloud 1026 may be integrated with the application coordination system 1028 that utilizes multiple GPUs to achieve seamless scaling and load balancing between and within the applications and services 920. In at least one embodiment, as described herein, the cloud 1026 may be responsible for performing at least some of the services 920 of the system 1000, including the compute service data 1666, the AI service data 1668, and / or the visualization service 1020. In at least one embodiment, the cloud 1026 may perform inference on large and small batches (e.g., execute NVIDIA's TENSOR RT), provide an accelerated parallel computing API and platform 1030 (e.g., NVIDIA's CUDA), execute the application coordination system 1028 (e.g., KUBERNETES), provide a graphics rendering API and platform (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher quality cinematic effects), and / or may provide other functions for the system 1000.

[0141] In at least one embodiment, to protect patient confidentiality (e.g., in the case of off-site use of patient data or records), cloud 1026 may include a registry - such as a deep learning container registry. In at least one embodiment, the registry may store containers for instantiating applications that may perform pre-processing, post-processing, or other processing tasks on patient data. In at least one embodiment, cloud 1026 may receive data including patient data and sensor data in the containers, perform the requested processing only on the sensor data in those containers, and then forward the resulting output and / or visualization to appropriate parties and / or devices (e.g., local medical devices for visualization or diagnosis) without extracting, storing, or otherwise accessing the patient data. In at least one embodiment, patient data confidentiality is maintained in accordance with HIPAA and / or other data regulations.

[0142] At least one embodiment of the present disclosure may be described according to the following clauses:

[0143] In clause 1, an image rendering method, the method comprising: identifying a first light in a scene, the first light having a first representation in a first color rendering scheme (CRS) and a second representation in a second CRS; identifying an attenuation function characterizing an interaction of the first light with a medium in the scene; using the attenuation function to determine a third representation of the first light interacting with the medium in the first CRS; determining one or more fitting parameters of the attenuation function at least in part based on minimizing a difference between the third representation of the first light and a fourth representation of the first light interacting with the medium in the second CRS; and rendering an image including at least one of the first light interacting with the medium or a second light interacting with the medium using the attenuation function and the determined one or more fitting parameters.

[0144] In clause 2, the method according to clause 1, wherein the first CRS is a spectral rendering scheme.

[0145] In clause 3, the method according to clause 1, wherein the second CRS is a trichromatic rendering scheme.

[0146] In clause 4, the method according to clause 1, wherein the attenuation function is an exponential function that is a product of an attenuation coefficient and a depth of the medium.

[0147] In clause 5, the method according to clause 1, wherein the attenuation function is a piecewise constant function of a wavelength including a plurality of values, each of the plurality of values being associated with a corresponding one of a plurality of wavelength intervals.

[0148] In clause 6, according to the method of clause 5, wherein one or more of the plurality of values are based on the attenuation coefficient of the medium for the respective color of the second CRS.

[0149] In clause 7, according to the method of clause 5, wherein the one or more fitting parameters include at least one boundary between adjacent intervals of the plurality of wavelength intervals.

[0150] In clause 8, according to the method of clause 1, wherein determining the one or more fitting parameters of the attenuation function comprises: identifying a plurality of chromaticity values corresponding to the third representation of the first light; and determining a metric representing the distance between each of the plurality of chromaticity values and a corresponding one of a plurality of reference chromaticity values.

[0151] In clause 9, according to the method of clause 8, wherein the plurality of reference chromaticity values are associated with a plurality of primary colors of the second CRS.

[0152] In clause 10, according to the method of clause 1, wherein the second light has a spectral composition different from that of the first light.

[0153] In clause 11, according to the method of clause 1, wherein the first light has a spectral composition of a natural outdoor light source.

[0154] In clause 12, a system comprising: a storage device; and one or more processing devices communicatively coupled to the storage device for: identifying a first light in a scene, the first light having a first representation in a first color rendering scheme (CRS) and a second representation in a second CRS; identifying an attenuation function characterizing the interaction of the first light with a medium in the scene; using the attenuation function to determine a third representation of the first light interacting with the medium in the first CRS; determining one or more fitting parameters of the attenuation function at least in part by minimizing a difference between the third representation of the first light and a fourth representation of the first light interacting with the medium in the second CRS; and using the attenuation function and the determined one or more fitting parameters to render an image including at least one of the first light interacting with the medium or a second light interacting with the medium.

[0155] In clause 13, according to the system of clause 12, wherein the first CRS is a spectral rendering scheme and the second CRS is a trichromatic rendering scheme.

[0156] In clause 14, according to the system of clause 12, wherein the attenuation function is an exponential function of the product of an attenuation coefficient and the depth of the medium.

[0157] In clause 15, the system according to clause 12, wherein the attenuation function is a piecewise constant function of wavelength including a plurality of values, each of the plurality of values being associated with a respective one of a plurality of wavelength intervals, and wherein one or more of the plurality of values are based on the attenuation coefficient of the medium for the respective color of the second CRS.

[0158] In clause 16, the system according to clause 15, wherein the one or more fitting parameters include at least one boundary between adjacent intervals of the plurality of wavelength intervals.

[0159] In clause 17, the system according to clause 12, wherein, to determine the one or more fitting parameters of the attenuation function, the one or more processing devices are further configured to: identify a plurality of chromaticity values corresponding to the third representation of the first light; and determine a metric representing the distance between each of the plurality of chromaticity values and a respective one of a plurality of reference chromaticity values.

[0160] In clause 18, the system according to clause 17, wherein the plurality of reference chromaticity values are associated with a plurality of primary colors of the second CRS.

[0161] In clause 19, the system according to clause 12, wherein the second light has a spectral composition different from the spectral composition of the first light.

[0162] In clause 20, a non-transitory computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processing device, cause the processing device to: identify a first light in a scene, the first light having a first representation in a first color rendering scheme (CRS) and a second representation in a second CRS; identify an attenuation function characterizing an interaction of the first light with a medium in the scene; use the attenuation function to determine a third representation of the first light interacting with the medium in the first CRS; determine one or more fitting parameters of the attenuation function at least in part by minimizing a difference between the third representation of the first light and a fourth representation of the first light interacting with the medium in the second CRS; and use the attenuation function and the determined one or more fitting parameters to render an image including at least one of the first light interacting with the medium or a second light interacting with the medium.

[0163] Other variations are within the spirit of the present disclosure. Thus, while the disclosed techniques are susceptible to various modifications and alternative constructions, certain of its embodiments have been shown in the drawings and have been described in detail above. However, it is to be understood that the intention is not to limit the disclosure to the one or more specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the present disclosure as defined by the appended claims.

[0164] Unless otherwise stated or clearly contradicted by the context, in the context of describing the disclosed embodiments (especially in the context of the appended claims), the use of the terms "a", "an", "the", and similar referents should be construed to cover both the singular and the plural, rather than as a definition of the terms. Unless otherwise stated, the terms "comprising", "having", "including", and "containing" should be construed as open-ended terms (meaning "including but not limited to"). The term "connected" (when not otherwise modified, referring to a physical connection) should be construed to mean included in whole or in part, attached to, or joined together, even with some intervening elements. Unless otherwise indicated herein, references to numerical ranges in this document are merely intended as a shorthand method for referring separately to each individual value falling within the range, and each individual value is incorporated into the specification as if it were recited herein individually. In at least one embodiment, unless otherwise indicated or clearly contradicted by the context, the use of the term "set" (e.g., "set of items") or "subset" should be construed to mean a non-empty set including one or more members. Additionally, unless otherwise indicated or clearly contradicted by the context, a "subset" of a corresponding set does not necessarily denote a proper subset of the corresponding set, but rather the subset and the corresponding set may be equal.

[0165] Unless otherwise expressly indicated or clearly contradicted by the context, conjunctive language such as the phrase “at least one of A, B, and C” or “at least one of A, B or C” is understood in context to mean items, clauses, etc., that can be A or B or C, or any non-empty subset of the set A and B and C. For example, in an illustrative example of a set having three members, the conjunctive phrases “at least one of A, B, and C” and “at least one of A, B or C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctive language is not generally intended to imply that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise specified or contradicted by the context, the term “plurality” indicates a plural state (e.g., “a plurality of items” means multiple items). In at least one embodiment, the number of items in a plurality of items is at least two, but can be more if expressly indicated or indicated by the context. Further, unless otherwise specified or clear from the context, the phrase “based on” means “at least partially based on” rather than “based solely on”.

[0166] Unless otherwise indicated herein or clearly contradicted by context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations and / or combinations thereof) are performed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that execute jointly on one or more processors by hardware or a combination thereof. In at least one embodiment, the code is stored, for example, in the form of a computer program on a computer-readable storage medium, the computer program including a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium, which excludes transitory signals (e.g., propagating transient electrical or electromagnetic transmissions), but includes non-transitory data storage circuits (e.g., buffers, caches, and queues) within a transient signal transceiver. In at least one embodiment, the code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which the executable instructions are stored, and when executed by one or more processors of a computer system (i.e., as a result of being executed), cause the computer system to perform the operations described herein. In at least one embodiment, a set of non-transitory computer-readable storage media includes a plurality of non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media among the plurality of non-transitory computer-readable storage media lack all of the code, but the plurality of non-transitory computer-readable storage media together store all of the code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors, e.g., the non-transitory computer-readable storage medium stores the instructions, and a main central processing unit (“CPU”) executes some instructions, while a graphics processing unit (“GPU”) and / or a data processing unit (“DPU”) (which may be together with the GPU) execute other instructions. In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of the instructions.

[0167] Thus, in at least one embodiment, a computer system is configured to implement one or more services that perform the operations of the processes described herein either individually or jointly, and such a computer system is configured with suitable hardware and / or software enabling the implementation of the operations. Additionally, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment is a distributed computer system that includes a plurality of devices operating in different ways such that the distributed computer system performs the operations described herein and such that a single device does not perform all of the operations.

[0168] Any and all uses of examples or exemplary language provided herein (e.g., "such as") are for the purpose of better illustrating embodiments of the present disclosure only and do not limit the scope of the disclosure, unless otherwise required. No language in the specification should be construed as indicating that any non-claimed element is essential for the practice of the disclosure.

[0169] All references cited herein, including publications, patent applications, and patents, are incorporated herein by reference to the extent as if each reference was individually and specifically indicated to be incorporated by reference and its entire content was set forth herein.

[0170] In the specification and claims, the terms "coupled" and "connected" and their derivatives may be used. It should be understood that these terms are not intended as synonyms for each other. Instead, in a particular example, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0171] Unless otherwise expressly stated, it is understood that throughout the specification, terms such as "processing", "computing", "calculating", "determining", etc., refer to actions and / or processes of a computer or computing system or similar electronic computing device that processes and / or transforms data represented as a physical quantity (e.g., electrons) in the registers and / or memory of the computing system into other data similarly represented as a physical quantity in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0172] In a similar manner, the term "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory and transforms that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" may be a CPU or a GPU. A "computing platform" may include one or more processors. As used herein, a "software" process may include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Similarly, each process may refer to multiple processes that execute instructions sequentially or in parallel, continuously or intermittently. In at least one embodiment, the terms "system" and "method" may be used interchangeably herein, provided that the system can embody one or more methods and the method can be considered a system.

[0173] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data may be accomplished in a variety of ways, such as by receiving data as a parameter of a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data may be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data may be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transferring, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transferring, sending, or presenting analog or digital data may be implemented by transmitting the data as an input or output parameter of a function call, an application programming interface, or a parameter of an interprocess communication mechanism.

[0174] Although the description herein sets forth example embodiments of the described technology, other architectures may be used to implement the described functionality and are intended to fall within the scope of the present disclosure. Additionally, although specific assignments of responsibilities are defined above for purposes of description, the various functions and responsibilities may be assigned and divided in different ways depending on the circumstances.

[0175] Moreover, although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter claimed in the appended claims need not be limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claims.

Claims

1. An image rendering method, the method comprising: Identifying a first light in a scene, the first light having a first representation in a first color rendering scheme (CRS) and a second representation in a second CRS; Identifying an attenuation function characterizing an interaction of the first light with a medium in the scene; Using the attenuation function to determine a third representation of the first light interacting with the medium in the first CRS; Determining one or more fitting parameters of the attenuation function based at least in part on minimizing a difference between the third representation of the first light and a fourth representation of the first light interacting with the medium in the second CRS; And Rendering an image including at least one of the first light interacting with the medium or a second light interacting with the medium using the attenuation function and the determined one or more fitting parameters.

2. The method according to claim 1, wherein the first CRS is a spectral rendering scheme.

3. The method according to claim 1, wherein the second CRS is a trichromatic rendering scheme.

4. The method according to claim 1, wherein the attenuation function is an exponential function that is a product of an attenuation coefficient and a depth of the medium.

5. The method according to claim 1, wherein the attenuation function is a piecewise constant function of wavelengths including a plurality of values, each of the plurality of values being associated with a corresponding one of a plurality of wavelength intervals.

6. The method according to claim 5, wherein one or more of the plurality of values are based on an attenuation coefficient of the medium for a corresponding color of the second CRS.

7. The method according to claim 5, wherein the one or more fitting parameters include at least one boundary between adjacent intervals of the plurality of wavelength intervals.

8. The method according to claim 1, wherein determining the one or more fitting parameters of the attenuation function comprises: Identifying a plurality of chromaticity values corresponding to the third representation of the first light; And Determining a metric representing a distance between each of the plurality of chromaticity values and a corresponding one of a plurality of reference chromaticity values.

9. The method according to claim 8, wherein the plurality of reference chromaticity values are associated with a plurality of primary colors of the second CRS.

10. The method according to claim 1, wherein the second light has a spectral composition different from that of the first light.

11. The method according to claim 1, wherein the first light has a spectral composition of a natural outdoor light source.

12. A system, comprising: A storage device; And One or more processing devices communicatively coupled to the storage device for: Identifying a first light in a scene, the first light having a first representation in a first color rendering scheme (CRS) and a second representation in a second CRS; Identifying an attenuation function characterizing an interaction of the first light with a medium in the scene; Using the attenuation function to determine a third representation of the first light interacting with the medium in the first CRS; Determine one or more fitting parameters of the attenuation function based at least in part on minimizing a difference between the third representation of the first light and the fourth representation of the first light interacting with the medium in the second CRS; And Use the attenuation function and the determined one or more fitting parameters to render an image including at least one of the first light interacting with the medium or the second light interacting with the medium.

13. The system according to claim 12, wherein the first CRS is a spectral rendering scheme and the second CRS is a trichromatic rendering scheme.

14. The system according to claim 12, wherein the attenuation function is an exponential function of the product of the attenuation coefficient and the depth of the medium.

15. The system according to claim 12, wherein the attenuation function is a piecewise constant function of wavelengths including a plurality of values, each of the plurality of values being associated with a respective one of a plurality of wavelength intervals, and wherein one or more of the plurality of values are based on the attenuation coefficient of the medium for the respective color of the second CRS.

16. The system according to claim 15, wherein the one or more fitting parameters include at least one boundary between adjacent intervals of the plurality of wavelength intervals.

17. The system according to claim 12, wherein, in order to determine the one or more fitting parameters of the attenuation function, the one or more processing devices are further configured to: Identify a plurality of chromaticity values corresponding to the third representation of the first light; and Determine a metric representing the distance between each of the plurality of chromaticity values and a respective one of a plurality of reference chromaticity values.

18. The system according to claim 17, wherein the plurality of reference chromaticity values are associated with a plurality of primary colors of the second CRS.

19. The system according to claim 12, wherein the second light has a spectral composition different from that of the first light.

20. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processing device, cause the processing device to: Identify a first light in a scene, the first light having a first representation in a first color rendering scheme (CRS) and a second representation in a second CRS; Identify an attenuation function characterizing the interaction of the first light with a medium in the scene; Use the attenuation function to determine a third representation of the first light interacting with the medium in the first CRS; Determine one or more fitting parameters of the attenuation function based at least in part on minimizing a difference between the third representation of the first light and the fourth representation of the first light interacting with the medium in the second CRS; And Use the attenuation function and the determined one or more fitting parameters to render an image including at least one of the first light interacting with the medium or the second light interacting with the medium.

Images