Automatically generated shader mask and parameters

Abstract

This invention provides a system, method, and storage medium for automatically generating shader masks and parameters. [Solution] The graphics shader 200 takes in two grayscale images (referred to as "masks") 300 and 302 and four colors 304, 306, 308, and 310 and generates a full-color image 312 from the input. The four colors 304, 306, 308, and 310 are, for example, red, green, blue, and yellow. The logic behind shader 200 allows for a wide variety of images by separating the colors from the image. (For example, by changing one color, a brick wall of different colors can be obtained). The two grayscale masks occupy less memory space than the full-color image.

Description

Technical Field

[0001] This application generally relates to automatically generated shader masks and parameters.

Background Art

[0002] As understood herein, computer simulations such as computer games use shaders, which are software programs, to fill game objects with color and texture. Also, as understood herein, as game graphics become more advanced, the memory space and processing time for shading operations become increasingly important.

Summary of the Invention

[0003] Thus, the device includes at least one computer storage that is not a temporary signal, and the at least one computer storage includes instructions executable by at least one processor to receive a first grayscale image and a second grayscale image and combine the grayscale images with a plurality of colors to render a test image. The instructions are executable to modify the test image using gradient descent and output a final color image based at least in part on a loss indication associated with the gradient descent.

[0004] The first grayscale image and the second grayscale image may be based on a common image, i.e., they may be two different grayscale versions of the same image. Alternatively, the first grayscale image and the second grayscale image may not be based on a common image.

[0005] In some embodiments, the instruction may be capable of rendering a test image by combining a grayscale image with four colors. The instruction may be embodied in a machine learning (ML) model and / or shader. In a non-limiting embodiment, the instruction may be capable of outputting a final color image to a computer simulation for display during playback of the computer simulation.

[0006] In another embodiment, the apparatus includes at least one processor, which identifies at least a first grayscale image and a second grayscale image, and is programmed with instructions to render a test image by combining the grayscale images with at least one color. The instructions are capable of modifying the test image using gradient descent and outputting a final color image, at least in part, based on loss indications associated with gradient descent.

[0007] In another embodiment, the method includes receiving a first grayscale image, receiving a second grayscale image, and receiving at least one color. The method includes outputting a test image based at least partially on the grayscale images and the color. The method further includes applying gradient descent to modify the test image by minimizing a loss function until a final image is generated, and outputting the final image to a computer simulation.

[0008] Details of this application, both in terms of its structure and operation, can best be understood by referring to the attached drawings, in which similar reference numerals refer to similar parts. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram of an exemplary system based on the principle of the present invention. [Figure 2] An exemplary hardware architecture is shown. [Figure 3] An example software architecture is shown. [Figure 4] This illustrates exemplary logic consistent with the principles of the present invention. [Figure 4A] Offer alternative expressions. [Figure 5] A screenshot illustrating the principle of this invention is provided. [Figure 6] A screenshot illustrating the principle of this invention is provided. [Figure 7] A screenshot illustrating the principle of this invention is provided. [Modes for carrying out the invention]

[0010] This disclosure generally relates to computer ecosystems including, but not limited to, forms of consumer electronic product (CE) device networks such as computer game networks. The systems herein may include server components and client components, which may be connected via a network to enable data exchange between the client components and the server components. Client components may include one or more computing devices, which may include game consoles such as Sony PlayStation®, or game consoles manufactured by Microsoft, Nintendo, or other manufacturers, extended reality (XR) headsets such as virtual reality (VR) headsets and augmented reality (AR) headsets, portable computers such as portable televisions (e.g., smart TVs, internet-enabled TVs), laptops, and tablet computers, and other mobile devices, including smartphones and additional examples described below. These client devices may operate in a variety of operating environments. For example, some of the client computers may employ, for example, a Linux® operating system, a Microsoft operating system, or a Unix® operating system, or an operating system manufactured by Apple or Google, or a BSD OS including a derivative of Berkeley Software Distribution (BSD). Using these operating environments, one or more browsing programs may be executed, such as browsers from Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted by the Internet servers described later. Furthermore, one or more computer game programs may be executed using the operating environment based on the principles of the present invention.

[0011] A server and / or gateway may be used, which may include one or more processors, one or more of which execute instructions that configure the server to send and receive data over a network such as the Internet. Alternatively, the client and server may be connected via a local intranet or a virtual private network. The server or controller may be instantiated by a game console such as Sony PlayStation®, a personal computer, etc.

[0012] Information can be exchanged between a client and a server over a network. For this purpose and for security, the server and / or client may include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form a device that implements a method of providing a secure community to network members, such as an online social website or a gamer network.

[0013] A processor can be a single-chip or multi-chip processor capable of executing logic through various lines such as address lines, data lines, and control lines, as well as registers and shift registers. A processor including a digital signal processor (DSP) can be a circuit embodiment.

[0014] Components included in one embodiment may be used in any preferred combination in other embodiments. For example, any of the various components described herein and / or depicted in the figures may be combined, replaced, or excluded from other embodiments.

[0015] "A system having at least one of A, B, and C" (similarly, "a system having at least one of A, B, or C" and "a system having at least one of A, B, and C") includes systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B and C together.

[0016] Referring here to Figure 1, an exemplary system 10 is shown, which may include one or more exemplary devices according to the principles of the present invention, as mentioned above and further described below. A first example of the exemplary devices included in system 10 is a consumer electronic product (CE) device such as an audio-video device (AVD) 12, which may include, but is not limited to, a projector-based theater display system or an internet-enabled TV with a TV tuner (equivalently, a set-top box that controls the TV). Alternatively, the AVD 12 may also be a computer-controlled internet-enabled ("smart") phone, a tablet computer, a notebook computer, a head-mounted device (HMD) and / or a headset such as smart glasses or a VR headset, other computer-controlled wearable devices, a computer-controlled internet-enabled music player, a computer-controlled internet-enabled headphones, an implantable skin device, and other computer-controlled internet-enabled implantable devices. In any case, it should be understood that AVD12 is configured to implement the principles of the present invention (for example, to communicate with other CE devices to implement the principles of the present invention, to execute the logic described herein, and to perform any other functions and / or operations described herein).

[0017] Therefore, to implement the principles of the present invention, the AVD12 can be established by some or all of the components shown. For example, the AVD12 may include one or more touch-enabled displays 14, which may be implemented as high-resolution or ultra-high-resolution flat screens of "4K" or higher. The touch-enabled display(s) 14 may include, for example, a capacitive or resistive touch-sensing layer with a touch-sensing electrode grid in accordance with the principles of the present invention.

[0018] The AVD12 may also include one or more speakers 16 for outputting audio in accordance with the principles of the present invention, and at least one additional input device 18, such as an audio receiver / microphone, for inputting audible commands to the AVD12 and controlling the AVD12. The exemplary AVD12 may also include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, WAN, LAN, etc., under the control of one or more processors 24. Thus, the interface 20 may, but is not limited to, a Wi-Fi transceiver, which is an embodiment of a wireless computer network interface, such as a mesh network transceiver. It should be understood that the processor 24 controls the AVD12 to carry out the principles of the present invention, and this includes other elements of the AVD12 as described herein, such as controlling a display 14 to present an image and receiving input from the display 14. Furthermore, it should be noted that the network interface 20 may be a wired or wireless modem or router, or may be another suitable interface such as a wireless telephone transceiver or the Wi-Fi transceiver described above.

[0019] In addition to the above, the AVD12 may also include one or more input and / or output ports 26, such as a High-Definition Multimedia Interface (HDMI®) port or a Universal Serial Bus (USB) port for physically connecting to another CE device, and / or a headphone port for connecting headphones to the AVD12 to provide the user with audio from the AVD12 via headphones. For example, an input port 26 may be connected by wire or wirelessly to a cable source 26a or satellite source 26a of audio video content. Thus, source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, source 26a may be a game console or disc player containing content. If source 26a is implemented as a game console, it may include some or all of the components described below in relation to the CE device 48.

[0020] The AVD12 may further include one or more computer memory / computer-readable storage media 28, such as disk-based storage or solid-state storage that are not transient signals, which in some cases are embodied as standalone devices within the AVD chassis, or as personal video recording devices (PVRs) or video disc players inside or outside the AVD chassis for playing AV programs, or as removable storage media or servers as described below. In some embodiments, the AVD12 may also include a location receiver or locating receiver, which includes, but is not limited to, a cell phone receiver 30, a GPS receiver 30, and / or an altimeter 30, and is configured to receive geographic location information from a satellite or cell phone base station, provide the information to a processor 24, and / or work in conjunction with the processor 24 to determine the altitude at which the AVD12 is located.

[0021] Continuing the description of AVD12, in some embodiments, AVD12 may include one or more cameras 32, which may be digital cameras such as thermal imaging cameras, web cameras, IR sensors, event-based sensors, and / or cameras integrated into AVD12, and may be controllable by the processor 24 to collect photos / images and / or videos according to the principles of the present invention. AVD12 may also include a Bluetooth transceiver 34 and another NFC element 36 that communicate with other devices using Bluetooth (registered trademark) and / or near-field communication (NFC) technologies, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.

[0022] Furthermore, the AVD12 may include one or more auxiliary sensors 38 that provide input to the processor 24. For example, one or more of the auxiliary sensors 38 may include one or more pressure sensors that form the layer of the touch-enabled display 14 itself, and may be, but are not limited to, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive wire strain gauges, optical pressure sensors, electromagnetic pressure sensors, etc. Other examples of sensors include pressure sensors, motion sensors such as accelerometers, gyroscopes, and cyclometers, or magnetic sensors, infrared (IR) sensors, optical sensors, velocity and / or cadence sensors, event-based sensors, and gesture sensors (e.g., sensing gesture commands). Thus, the sensors 38 may be implemented by an inertial measurement unit (IMU) that typically includes a combination of accelerometers, gyroscopes, and magnetometers to determine the position and orientation of the AVD12 in three dimensions, or by an event-based sensor such as an event detection sensor (EDS). An EDS in accordance with this disclosure provides an output indicating a change in light intensity sensed by at least one pixel of a photosensing array. For example, if the light sensed by the pixel is decreasing, the output of the EDS may be -1, and if it is increasing, the output of the EDS may be +1. If the output binary signal is 0, it may indicate that there is no change in light intensity below a certain threshold.

[0023] The AVD12 may also include an OTA TV broadcast port 40 for receiving over-the-air (OTA) TV broadcasts that provide an input to the processor 24. In addition to the above, it should be noted that the AVD12 may also include an infrared (IR) transmitter 42 and / or an IR receiver 42 and / or an IR transceiver 42, such as an Infrared Data Association (IRDA) device. A battery (not shown) may be provided to power the AVD12, which may be a kinetic energy harvester that can convert kinetic energy into electricity to charge the battery and / or power the AVD12. A graphics processing unit (GPU) 44 and a field programmable gate array 46 may also be included. One or more tactile / vibration generators 47 may be provided to generate tactile signals that can be sensed by a person holding or in contact with the device. Thus, the tactile generator 47 may use an electric motor connected to an eccentric weight and / or an unbalanced weight to vibrate the whole or part of the AVD12 via the rotatable shaft of the motor, whereby, as the shaft rotates under the control of the motor (the motor may be controlled by a processor such as the processor 24), vibrations of various frequencies and / or amplitudes, as well as simulations of forces in various directions, can be produced.

[0024] Light sources such as projectors, such as an infrared (IR) projector, may also be included.

[0025] In addition to AVD12, System 10 may include one or more other CE device types. In one embodiment, the first CE device 48 may be a computer game console that can be used to transmit audio and video of a computer game to AVD12 via commands sent directly to AVD12 and / or sent through a server as described below, while the second CE device 50 may include components similar to the first CE device 48. In the shown embodiment, the second CE device 50 may be configured as a computer game controller operated by a player, or as a head-mounted display (HMD) worn by a player. The HMD may include a head-up transparent or opaque display that presents AR / MR content or VR content (more commonly, Extended Reality (XR) content), respectively. The HMD may be configured as a glasses-type display or as a large VR-type display sold by a computer game equipment manufacturer.

[0026] In the embodiments shown, only two CE devices are shown, but it will be understood that fewer or more devices may be used. The devices herein may implement some or all of the components shown with respect to AVD12. Any of the components shown in the figures below may incorporate some or all of the components shown in the AVD12 example.

[0027] Referring here to the at least one server 52 described above, the at least one server 52 includes at least one server processor 54, at least one tangible computer-readable storage medium 56 such as disk-based storage or solid-state storage, and at least one network interface 58, the at least one network interface 58 enabling communication with other exemplary devices over the network 22 under the control of the server processor 54, and can actually facilitate communication between the server and client devices in accordance with the principles of the present invention. It should be noted that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface such as a wireless telephone transceiver.

[0028] Therefore, in some embodiments, server 52 may be an internet server or an entire server "farm," and may include and run "cloud" functionality, such as in exemplary embodiments like a network game application, that allows devices of system 10 to access the "cloud" environment via server 52. Alternatively, server 52 may be implemented by one or more game consoles or other computers in the same room or nearby as the other illustrated devices.

[0029] The components shown in the diagram below may include some or all of the components shown herein. Any user interface (UI) described herein may be integrated and / or extended, and UI elements may be mixed and matched between UIs.

[0030] The principles of the present invention can utilize a variety of machine learning models, including deep learning models. Machine learning models in accordance with the principles of the present invention can utilize a variety of algorithms trained in ways including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms that can be implemented by computer circuits include one or more neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and a type of RNN known as a long-short-term memory (LSTM) network. Support vector machines (SVMs) and Bayesian networks can also be considered examples of machine learning models. In addition to the types of networks described above, the models herein can be implemented by classifiers.

[0031] Therefore, as understood herein, performing machine learning may involve accessing a model, training the model with training data, and enabling the model to process further data and perform inference. Thus, an artificial neural network / artificial intelligence model trained through machine learning may include an input layer, an output layer, and multiple hidden layers between them, the multiple hidden layers being configured and weighted to perform inference about a preferred output.

[0032] Figure 2 shows a graphics shader 200 that can be executed by the GPU 202, which shades graphics from a source of computer game graphics 204, such as a computer game console or server, for presentation on the display 206.

[0033] As shown in Figure 3, shader 200 takes in two grayscale images (referred to as "masks") 300 and 302 and four colors 304, 306, 308, and 310 to generate a full-color image 312 from the input. The four colors 304, 306, 308, and 310 could be, for example, red, green, blue, and yellow. The logic behind shader 200 consists of two parts. Firstly, by separating the colors from the image, a wide variety of images become possible (for example, by changing one color, a brick wall of different colors can be obtained), and secondly, the two grayscale masks occupy less memory space than the full-color image.

[0034] A script using differentiable programming and gradient descent "finds" the mask and color of a target image. Figure 4 shows exemplary logic that such a script may implement.

[0035] Starting with block 400, two grayscale masks 300 and 302, shown in Figure 3, are generated along with four colors 304, 306, 308, and 310, also shown in Figure 3.

[0036] Proceeding to block 402, the logic generates a color image based on the grayscale masks 300, 302 and the four colors 304, 306, 308, and 310. Next, proceeding to block 404, the loss between the current image and a target image that may be defined by the user is identified. In the determination rhombus 406, if the identified loss is small enough to be acceptable, or if a predetermined number of iterations has been reached, then in block 410, the final grayscale mass and color are output. However, if the identified loss is not small enough to be acceptable, or if a predetermined number of iterations has not been reached, the logic proceeds from the determination rhombus 406 to block 408.

[0037] In block 408, the grayscale mask and color are corrected by applying gradient descent, and the logic loops back to block 402 to generate the updated image.

[0038] Gradient descent uses calculus to take a loss as input and identify a way to modify the image to reduce the loss. This technique can be used in stochastic gradient descent and can be used as an extension of the backpropagation algorithm used to train ML models, such as the ML models described herein. Stochastic gradient descent adds a stochastic characteristic in the direction of the update. Weights may be used to compute the derivative.

[0039] Figure 4A shows that image parameters are generated in block 420. The parameters can consist of two grayscale masks and four color pixel values, represented by three values ​​for each RGB image. Therefore, if the grayscale mask is 2000 × 20000 pixels, the total number of parameters is 2000 × 2000 × 2 + 4 × 3 parameters. Generating an image from the parameters means performing the same calculations that the shader does. In the case of pseudocode: a=lerp(color_0, color_1, mask_0) b=lerp(color_2, color_3, mask_0) Generated_image=lerp(a, b, mask_1)

[0040] In block 422, the loss is obtained. The loss is the difference between the generated image and the target image (the difference is either the absolute value or the mean squared error). Next, in block 424, the loss is backpropagated using gradient descent to correct the parameters of block 420. The user can determine the number of cycles to repeat, for example, 1000 to 6000 cycles.

[0041] This is further illustrated by Figures 5 and 6. In Figure 5, two grayscale masks 500 and 502 are applied to the previous result and another mask to render a final color image 504, which is enlarged to 506. In Figure 6, two colors 600 and 602 are combined with the final color 504 from Figure 5 to render a new color image 604.

[0042] Figure 7 shows two grayscale masks 700 and 702, which are essentially two different grayscale versions of the same image, and are combined with four colors 704 (red, black, yellow, and blue in the shown embodiment) to render the final color image 712.

[0043] The tools and techniques described above may be provided to end-user game computing devices such as computer game consoles, thereby enabling end-user game players to use the tools described herein within a game (i.e., as part of playing a computer game) to create and / or modify game objects with one another.

[0044] While specific embodiments are shown and described in detail in this specification, it should be understood that the subject matter included in the present invention is limited only by the claims.

Claims

1. A system comprising one or more storage media for storing instructions and one or more processors configured to execute the instructions, The aforementioned processor executes the instruction, Receiving a first grayscale image, a second grayscale image, and a color set, The first grayscale image, the second grayscale image, and the color set are combined to render a color image. Outputting the aforementioned color image, A system that causes the system to perform an operation including the following.

2. The system according to claim 1, wherein the first grayscale image is based on the first image, and the second grayscale image is based on the first image.

3. The system according to claim 1, wherein the first grayscale image is based on the first image, and the second grayscale image is based on the second image.

4. The system according to claim 1, wherein the color set includes multiple colors.

5. The system according to claim 1, wherein the color image includes the color set.

6. The system according to claim 1, wherein the first file size of the first grayscale image is smaller than the second file size of the color image.

7. The system according to claim 1, wherein the sum of the first file size of the first grayscale image and the second file size of the second grayscale image is smaller than the third file size of the color image.

8. Receiving a first grayscale image, a second grayscale image, and a color set, The first grayscale image, the second grayscale image, and the color set are combined to render a color image. Outputting the aforementioned color image, Methods that include...

9. The method according to claim 8, wherein the first grayscale image is based on the first image, and the second grayscale image is based on the first image.

10. The method according to claim 8, wherein the first grayscale image is based on the first image, and the second grayscale image is based on the second image.

11. The method according to claim 8, wherein the color set includes multiple colors.

12. The method according to claim 8, wherein the color image includes the color set.

13. The method according to claim 8, wherein the first file size of the first grayscale image is smaller than the second file size of the color image.

14. The method according to claim 8, wherein the sum of the first file size of the first grayscale image and the second file size of the second grayscale image is smaller than the third file size of the color image.

15. One or more non-temporary computer-readable storage media for storing instructions, When the aforementioned instruction is executed by one or more processors of the system, Receiving a first grayscale image, a second grayscale image, and a color set, The first grayscale image, the second grayscale image, and the color set are combined to render a color image. Outputting the aforementioned color image, A non-temporary, computer-readable storage medium that causes the system to perform an operation including the following.

16. The non-temporary computer-readable storage medium according to claim 15, wherein the first grayscale image is based on the first image, and the second grayscale image is based on the first image.

17. The non-temporary computer-readable storage medium according to claim 15, wherein the first grayscale image is based on the first image and the second grayscale image is based on the second image.

18. The non-temporary computer-readable storage medium according to claim 15, wherein the color set includes multiple colors.

19. The non-temporary computer-readable storage medium according to claim 15, wherein the first file size of the first grayscale image is smaller than the second file size of the color image.

20. The non-temporary computer-readable storage medium according to claim 15, wherein the sum of the first file size of the first grayscale image and the second file size of the second grayscale image is smaller than the third file size of the color image.

Images