Automatically generated shader masks and parameters

Abstract

A graphics shader (200) receives two grayscale images (referred to as "masks") (300, 302) and four colors, and generates a full color image. The logic behind the shader is dual. First, the colors are separated (408) from the image to achieve more diversified images (e.g., a different color of brick walls can be obtained by changing one color), and second, the two gray masks occupy less memory space than a full color image. The mask and color of the target image are "found" using a differentially programmable and gradient descent script.

Description

Technical Field

[0001] The present application generally relates to automatically generated shader masks and parameters. Background Art

[0002] As understood herein, computer simulations (such as computer games) use shaders (which are software programs) to fill game objects with colors and textures. As also understood herein, as game graphics become more complex, the memory space and processing time of shading operations become increasingly important. Summary of the invention

[0003] Thus, an apparatus includes at least one computer storage device that is not a transient signal and further includes instructions executable by at least one processor to receive a first grayscale image and a second grayscale image and combine the grayscale images with a plurality of colors to present a test image. The instructions are executable to transform the test image using gradient descent and output a final color image based at least in part on a loss indication associated with the gradient descent.

[0004] The first grayscale image and the second grayscale image may be based on a common image, that is, may be two different grayscale versions of the same image. Alternatively, the first grayscale image and the second grayscale image may not be based on a common image.

[0005] In some examples, the instructions may be executable to combine the grayscale image with four colors to present the test image. The instructions may be embodied in a machine learning (ML) model and / or a shader. In a non-limiting embodiment, the instructions may be executable to output the final color image to a computer simulation so that the final color image is displayed during playback of the computer simulation.

[0006] In another aspect, an apparatus includes at least one processor programmed with instructions to identify at least a first grayscale image and a second grayscale image and combine the grayscale images with at least one color to present a test image. The instructions are executable to transform the test image using gradient descent and output a final color image based at least in part on a loss indication associated with the gradient descent.

[0007] In another aspect, a method includes receiving a first grayscale image, receiving a second grayscale image, and receiving at least one color. The method includes outputting a test image based at least in part on the grayscale image and the color. The method also includes applying gradient descent to minimize a loss function to change the test image until a final image is generated, and outputting the final image to a computer simulation.

[0008] The structural and operational details of the present application may be best understood with reference to the accompanying drawings, in which like reference numerals represent like parts, and in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram of an exemplary system according to the present principles;

[0010] Figure 2 An exemplary hardware architecture is shown;

[0011] Figure 3 An exemplary software architecture is shown;

[0012] Figure 4 An exemplary logic consistent with the present principles is shown;

[0013] Figure 4A Alternative representations are presented; and

[0014] Figures 5 to 7 The present principles are graphically illustrated in screenshot form. DETAILED DESCRIPTION

[0015] The present disclosure generally relates to computer ecosystems, including aspects of consumer electronics (CE) device networks, such as, but not limited to, computer gaming networks. The systems herein may include server and client components that may be connected via a network so that data may be exchanged between the client and server components. The client components may include one or more computing devices, including a gaming console (such as a Sony Play Station 3, or a gaming console such as a Sony Play Station 3). ® or game consoles manufactured by Microsoft or Nintendo or other manufacturers), extended reality (XR) head-mounted kits (such as virtual reality (VR) head-mounted kits, augmented reality (AR) head-mounted kits), portable televisions (e.g., smart TVs, Internet-enabled televisions), portable computers (such as laptops and tablet computers), and other mobile devices (including smartphones and other examples discussed below). These client devices can operate in various operating environments. For example, some of the client computers can use the Linux operating system, Microsoft's operating system, or the Unix operating system, or an operating system produced by Apple, Inc or Google, or the Berkeley Software Distribution or the Berkeley Standard Distribution (BSD) operating system (including descendants of BSD) as examples. These operating environments can be used to execute one or more browsing programs, such as browsers made by Microsoft or Google or Mozilla or other browser programs that can access websites hosted by Internet servers discussed below. In addition, an operating environment according to the present principles can be used to execute one or more computer game programs.

[0016] A server and / or gateway may be used, which may include one or more processors executing instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, the client and server may be connected via a local intranet or virtual private network. The server or controller may be powered by a game console such as a Sony PlayStation ® ), personal computers, etc.

[0017] Information may be exchanged between the client and the server over the network. To this end, for security purposes, the server and / or client may include firewalls, load balancers, temporary storage and proxies, and other network infrastructure to ensure reliability and security. One or more servers may form an apparatus that implements a method for providing a secure community (such as an online social networking site or a gamer network) to network members.

[0018] The processor may be a single chip processor or a multi-chip processor that may execute logic through various lines such as address lines, data lines, control lines, and registers and shift registers. A processor including a digital signal processor (DSP) may be an embodiment of the circuit.

[0019] The components included in one embodiment may be used in other embodiments in any appropriate combination. For example, any of the various components described herein and / or depicted in the drawings may be combined, interchanged or excluded from other embodiments.

[0020] “A system having at least one of A, B, and C” (similarly, “a system having at least one of A, B, or C” and “a system having at least one of A, B, C”) includes systems having only A, only B, only C, sometimes A and B, sometimes A and C, sometimes B and C, and / or sometimes A, B, and C.

[0021] Reference now Figure 1, an exemplary system 10 is shown, which may include one or more of the exemplary devices mentioned above and further described below according to the principles of the present invention. The first of the exemplary devices included in the system 10 is a consumer electronics (CE) device, such as an audio video device (AVD) 12, such as but not limited to a theater display system that can be based on a projector, or an Internet-enabled TV with a TV tuner (equivalent to a set-top box that controls the TV). Alternatively, the AVD 12 can also be a computerized Internet ("smart") phone, a tablet notebook, a notebook computer, a head-mounted device (HMD) and / or a head-mounted kit (such as smart glasses or a VR head-mounted kit), another wearable computerized device, a computerized Internet-enabled music player, a computerized Internet-enabled headset, a computerized Internet-enabled implantable device (such as an implantable skin device), etc. In any case, it should be understood that the AVD 12 is configured to perform the present principles (e.g., communicate with other CE devices to perform the present principles, perform the logic described herein, and perform any other functions and / or operations described herein).

[0022] Thus, to perform these remote operations, the AVD 12 may be built with some or all of the components shown. For example, the AVD 12 may include one or more touch displays 14, which may be implemented with a high-definition or ultra-high-definition "4K" or higher flat panel screen. The touch display 14 may include, for example, a capacitive or resistive touch sensing layer with an electrode grid for touch sensing consistent with the present principles.

[0023] AVD 12 may also include one or more speakers 16 for outputting audio in accordance with the present principles, and at least one additional input device 18, such as an audio receiver / microphone, for inputting audible commands to AVD 12 to control AVD 12. Example AVD 12 may also include one or more network interfaces 20 for communicating through at least one network 22 (such as the Internet, WAN, LAN, etc.) under the control of one or more processors 24. Thus, interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that processor 24 controls AVD 12 to perform the present principles, including other elements of AVD 12 described herein, such as controlling display 14 to present images thereon and receiving input therefrom. In addition, please note that network interface 20 may be a wired or wireless modem or router, or other suitable interface, such as a wireless telephone transceiver, or a Wi-Fi transceiver as mentioned above, etc.

[0024] In addition to the above, the AVD 12 may also include one or more input and / or output ports 26, such as a high-definition multimedia interface (HDMI) port or a universal serial bus (USB) port, to physically connect to another CE device and / or a headphone port to connect headphones to the AVD 12, thereby presenting audio from the AVD 12 to the user through the headphones. For example, the input port 26 may be connected to a cable or satellite source 26a of audio and video content via a wired or wireless method. Therefore, the source 26a may be a separate or integrated set-top box, or a satellite receiver. Alternatively, the source 26a may be a game console or disk player containing content. The source 26a may include some or all of the components described below about the CE device 48 when implemented as a game console.

[0025] The AVD 12 may also include one or more computer memories / computer readable storage media 28 that are not transient signals, such as disk-based storage devices or solid-state storage devices, which in some cases are embodied in the chassis of the AVD as stand-alone devices or personal video recording devices (PVRs) or video disk players, which may be located inside or outside the chassis of the AVD for playing AV programs, or may be removable storage media or servers as described below. In addition, in some embodiments, the AVD 12 may include a location or position receiver, such as, but not limited to, a cell phone receiver, a GPS receiver, and / or an altimeter 30, which is configured to receive geographic location information from a satellite or cell phone base station and provide the information to the processor 24 and / or determine, together with the processor 24, the altitude at which the AVD 12 is set.

[0026] Continuing with the description of the AVD 12, in some embodiments, the AVD 12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras (such as web cameras), infrared sensors, event-based sensors, and / or cameras integrated into the AVD 12 and controllable by the processor 24 to collect pictures / images and / or videos according to the present principles. The AVD 12 may also include a Bluetooth ® The transceiver 34 and other near field communication (NFC) elements 36 may be used to communicate with other devices using Bluetooth and / or NFC technology, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.

[0027] In addition, the AVD 12 may include one or more auxiliary sensors 38 that provide input to the processor 24. For example, one or more of the auxiliary sensors 38 may include one or more pressure sensors that form a layer of the touch-enabled display screen 14 itself, and may be, but are not limited to, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive strain gauges, optical pressure sensors, electromagnetic pressure sensors, and the like. Other sensor examples include pressure sensors, motion sensors (such as accelerometers, gyroscopes, cycle meters, or magnetic sensors), infrared (IR) sensors, optical sensors, speed and / or rhythm sensors, event-based sensors, gesture sensors (e.g., for sensing gesture commands). Thus, the sensor 38 may be implemented by one or more motion sensors, such as individual accelerometers, gyroscopes, and magnetometers, and / or an inertial measurement unit (IMU), which typically includes a combination of accelerometers, gyroscopes, and magnetometers to determine the position and orientation of the AVD 12 in three dimensions, or by an event-based sensor, such as an event detection sensor (EDS). An EDS consistent with the present disclosure provides an output that indicates a change in light intensity sensed by at least one pixel of a light sensing array. For example, if the light sensed by the pixel is decreasing, the output of the EDS may be -1; if the sensed light is increasing, the output of the EDS may be + 1. An output binary signal of 0 may indicate that the light intensity has not changed below a certain threshold.

[0028] The AVD 12 may also include a wireless TV broadcast port 40 for receiving OTA TV broadcasts that provide input to the processor 24. In addition to the above, it is worth noting that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided to power the AVD 12, and a kinetic energy collector may also be provided to convert kinetic energy into electricity to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field programmable gate array 46 may also be included. One or more tactile / vibration generators 47 may be provided to generate tactile signals that can be sensed by a person holding or contacting the device. Thus, the tactile generator 47 may vibrate all or part of the AVD 12 using an electric motor that is connected to an eccentric and / or unbalanced weight via a rotating shaft of the motor so that the shaft may rotate under the control of the motor (the motor may then be controlled by a processor (such as the processor 24)) to generate vibrations of various frequencies and / or amplitudes and force simulations in various directions.

[0029] A light source may also be included, such as a projector, such as an infrared (IR) projector.

[0030] In addition to the AVD 12, the system 10 may also include one or more other CE device types. In one example, the first CE device 48 may be a computer game console that can be used to send computer game audio and video to the AVD 12 via commands sent directly to the AVD 12 and / or through the server described below, while the second CE device 50 may include components similar to the first CE device 48. In the example shown, the second CE device 50 may be configured as a computer game controller manipulated by the player or a head-mounted display (HMD) worn by the player. The HMD may include a heads-up transparent or opaque display for presenting AR / MR content or VR content (more generally, extended reality (XR) content), respectively. The HMD may be configured as a glasses-type display, or as a larger VR-type display sold by a computer game device manufacturer.

[0031] In the example shown, only two CE devices are shown, but it should be understood that fewer or more devices may be used. The devices herein may implement some or all of the components shown for AVD 12. Any of the components shown in the following figures may include some or all of the components shown in the case of AVD 12.

[0032] Referring now to the aforementioned at least one server 52, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56 (such as a disk-based storage device or a solid-state storage device), and at least one network interface 58, which allows communication with other devices shown over the network 22 under the control of the server processor 54, and can actually facilitate communication between the server and client devices according to the present principles. Note that the network interface 58 can be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface (such as, for example, a wireless telephone transceiver).

[0033] Thus, in some embodiments, server 52 may be an Internet server or an entire server "farm" and may include and perform "cloud" functionality such that the devices of system 10 may access a "cloud" environment, such as for networked gaming applications, via server 52 in an example embodiment. Alternatively, server 52 may be implemented by one or more gaming consoles or other computers located in the same room or nearby with the other devices shown.

[0034] The components shown in the figure below may include some or all of the components shown in this article. Any user interface (UI) described in this article can be combined and / or extended, and UI elements can be mixed and matched between UIs.

[0035] The present principles may employ various machine learning models, including deep learning models. Machine learning models consistent with the present principles may use various algorithms trained in a manner including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms may be implemented by computer circuits, including one or more neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and a type of RNN called a long short-term memory (LSTM) network. Support vector machines (SVMs) and Bayesian networks may also be considered examples of machine learning models. In addition to the network types set forth above, the models herein may also be implemented by classifiers.

[0036] Thus, as understood herein, performing machine learning may involve accessing training data and subsequently training a model based on the training data to enable the model to process further data for inference. Thus, an artificial neural network / artificial intelligence model trained via machine learning may include an input layer, an output layer, and multiple hidden layers therebetween that are configured and weighted to infer appropriate outputs.

[0037] Figure 2 Graphics shader 200 is shown, which may be executed by GPU 202 for shading graphics from a computer game graphics source 204 , such as a computer game console or server, for presentation on display 206 .

[0038] like Figure 3 As shown, the shader 200 takes two grayscale images (called "masks") 300, 302 and four colors 304, 306, 308, 310 and generates a full-color image 312 from the input. The four colors 304, 306, 308, 310 can be, for example, red, green, blue, and yellow. The logic behind the shader 200 is twofold. First, separating the colors from the image allows for more diverse images (for example, changing one color can result in a different color of a brick wall), and second, the two grayscale masks take up less memory space than the full-color images.

[0039] Script to "find" the mask and color of a target image using differentiable programming and gradient descent. Figure 4 Exemplary logic that such a script may implement is shown.

[0040] Starting from block 400, generate Figure 3 The two grayscale masks 300, 302 and Figure 3 Four colors 304, 306, 308, 310 are shown.

[0041] Continuing to block 402, the logic generates a color image based on the grayscale masks 300, 302 and the four colors 304, 306, 308, 310. Next, proceeding to block 404, a loss between the current image and a target image is determined, which target may be user defined. Based on the determined loss, a decision diamond 406 indicates that if the determined loss is acceptably small, or a predetermined number of iterations has been reached, the final grayscale quality and color is output at block 410. However, if the determined loss is not acceptably small, or if a predetermined number of iterations has not been reached, the logic moves from the decision diamond 406 to block 408.

[0042] At block 408, gradient descent is applied to modify the grayscale mask and color, and the logic loops back to block 402 to generate an updated image.

[0043] Gradient descent uses calculus to receive the loss as input and determine how to modify the image to reduce the loss. This technique can be used with stochastic gradient descent and can be used as an extension of the backpropagation algorithm used to train ML models such as the one described in this article. Stochastic gradient descent adds randomness in the direction of the update. The weights can be used to calculate derivatives.

[0044] Figure 4A is a representation of the image parameters generated at block 420. The parameters can be two grayscale masks and pixel values ​​for four colors, with three values ​​for each RGB image. So if the grayscale mask is 2000x20000 pixels, the total number of parameters is 2000x2000x2 + 4x3 parameters. Generating an image from the parameters means performing the same calculations as the shader. The pseudo code is as follows:

[0045] a = lerp(color0,color1,mask0)

[0046] b = lerp(color2,color3,mask0)

[0047] Generated image = lerp(a,b,mask1)

[0048] Continuing to block 422, the loss is obtained. The loss is the difference between the generated image and the target image (the difference is either absolute value or mean squared error). Then, at block 424, the loss is back-propagated using gradient descent to modify the parameters at block 420. The user can determine the number of times the cycle is repeated, for example, one thousand to six thousand cycles.

[0049] Figure 5 and Figure 6 Further shown. Figure 5 In FIG. 5 , two grayscale masks 500 , 502 are applied to the previous results plus another mask to render a final color image 504 , shown enlarged at 506 . Figure 6 In the two colors 600, 602 and Figure 5 The final colors 504 in are combined to present a new color image 604.

[0050] Figure 7 Two grayscale masks 700 , 702 are shown, which are essentially two different grayscale versions of the same image, which are combined with four colors 704 (red, black, yellow and blue in the example shown) to render a final color image 712 .

[0051] The above-described tools and techniques may be provided in an end-user gaming computing device (such as a computer game console) such that end-user game players may use the tools described herein to create and / or modify game objects for each other in-game (i.e., as part of playing a computer game).

[0052] While particular embodiments have been herein shown and described in detail, it is to be understood that the subject matter encompassed by the present invention is limited only by the claims.

Claims

1. A device comprising: at least one computer storage device, the at least one computer storage device being not a transient signal and comprising instructions executable by at least one processor to: receiving a first grayscale image and a second grayscale image; combining the grayscale image with a plurality of colors to present a test image; transforming the test image using gradient descent; as well as A final color image is output based at least in part on a loss indication associated with the gradient descent. 2 . The apparatus of claim 1 , wherein the first grayscale image and the second grayscale image are based on a common image. 3 . The apparatus of claim 1 , wherein the first grayscale image and the second grayscale image are not based on a common image.

4. The apparatus of claim 1, wherein the instructions are executable to: The grayscale image is combined with four colors to present the test image.

5. The apparatus of claim 1, wherein the instructions are embodied in a machine learning (ML) model. The apparatus of claim 1 , wherein the instructions are embodied in a shader.

7. The apparatus of claim 1, wherein the instructions are executable to output the final color image to a computer simulation so that the final color image is displayed during playback of the computer simulation.

8. The apparatus of claim 1, comprising the at least one processor.

9. A device comprising: at least one processor programmed with instructions to: identifying at least a first grayscale image and a second grayscale image; combining the grayscale image with at least one color to present a test image; transforming the test image using gradient descent; as well as A final color image is output based at least in part on a loss indication associated with the gradient descent.

10. The apparatus of claim 9, wherein the first grayscale image and the second grayscale image are based on a common image.

11. The apparatus of claim 9, wherein the first grayscale image and the second grayscale image are not based on a common image.

12. The apparatus of claim 9, wherein the instructions are executable to: The grayscale image is combined with four colors to present the test image.

13. The apparatus of claim 9, wherein the instructions are embodied in a machine learning (ML) model.

14. The apparatus of claim 9, wherein the instructions are embodied in a shader.

15. The apparatus of claim 9, wherein the instructions are executable to output the final color image to a computer simulation so that the final color image is displayed during playback of the computer simulation.

16. A method comprising: receiving a first grayscale image; receiving a second grayscale image; Receive at least one color; outputting a test image based at least in part on the grayscale image and the color; Applying gradient descent to minimize a loss function to transform the test image until a final image is generated; as well as The final image was exported to a computer simulation. The method of claim 16 , wherein the first grayscale image and the second grayscale image are based on a common image. The method of claim 16 , wherein the first grayscale image and the second grayscale image are not based on a common image.

19. The method of claim 16, wherein the method is embodied in a machine learning (ML) model.

20. The method of claim 16, wherein the method is performed by a shader.

Images