Capturing computer game output as it is rendered for 2D to 3D conversion, accessibility, and other effects

Abstract

If the complete image is synthesized, important information for image generation, such as video generation for computer games, may be lost. To avoid this, a customized capture script (600) is created for a computer game that specifies (502) what to capture for future use (504) from the frame buffer (400), depth buffer (402), stencil buffer (404), etc., during the rendering of a game video frame. The script also specifies when data should be recorded during frame rendering. By capturing data mid-frame, on-screen object boundaries and relative distances between objects can be inferred. This information can be used (506) for 2D to 3D conversion, object coloring for increased contrast, object highlighting, etc.

Description

[Technical Field]

[0001] This application relates generally to capturing computer game output as it is being rendered for 2D to 3D conversion, accessibility, and other effects. [Background technology]

[0002] As understood herein, when rendering a video frame for a computer simulation such as a computer game, multiple layers may be rendered in which some data buffers are reused before completing composition of the full frame. For example, a color buffer may be used to render a layer of object shapes, a depth buffer may be used to render a layer representing the depth of the frame, and then other layers may be rendered, including lighting layers, transparent layers, opaque layers, etc., by reusing the color and depth buffers. Summary of the Invention

[0003] As will be further understood herein, if the operating system does not have access to, for example, a computer game running on the OS apart from the commands the game issues to a graphics processing unit (GPU) to render frames of video, information that may provide useful information to the OS for purposes described herein may be lost due to the reuse of buffers to render multiple layers of frames.

[0004] Thus, the device includes at least one computer storage, the at least one computer storage including instructions executable by at least one processor rather than transient signals, the instructions configuring a first video frame of the computer simulation at least in part by rendering a first video layer using a first data buffer, rendering a second video layer using the first data buffer, and capturing data in the first data buffer during rendering before completing compositing of the first video frame. The instructions are executable to modify an appearance of at least one object associated with the computer simulation using data captured from the first data buffer.

[0005] In an exemplary embodiment, the object is rendered in two dimensions (2D), the first data buffer includes a depth data buffer, and the instructions are executable to at least partially modify the appearance of the object by rendering the object in 3D in at least one frame following the first frame or in the first frame.

[0006] In another example, the objects are rendered in two dimensions (2D), the first data buffer includes a depth data buffer, and the instructions are executable to at least partially modify the appearance of the objects by rendering at least one object in 3D in at least one frame of the remastered version of the computer simulation.

[0007] In some implementations, the object is rendered in a first color in a first frame, the first data buffer includes a color data buffer, and the instructions are executable to at least partially modify the appearance of the object by presenting the object in a second color, different from the first color, in at least one frame following the first frame and / or in the first frame.

[0008] In another embodiment, the object is rendered in a first color in a first frame, the first data buffer includes a color data buffer, and the instructions are executable to at least partially modify the appearance of the object by presenting the object in a second color, different from the first color, in at least one frame of the remastered version of the computer simulation.

[0009] In an exemplary embodiment, the instructions are executable to at least partially change the appearance of the object by highlighting the object in at least one frame following the first frame. Note that additionally or alternatively, the first frame itself may be modified, altered / remastered prior to display.

[0010] In another exemplary embodiment, the instructions are executable to at least partially change the appearance of the object by highlighting the object in at least one frame of the remastered version of the computer simulation.

[0011] If desired, a script can be used to identify a first frame and / or a first data buffer to use in capturing data.

[0012] In another aspect, an apparatus includes at least one processor programmed with instructions for rendering at least one frame of a computer-simulated video, the instructions being executable to, during the rendering, record first data from at least one buffer used in rendering the frame, whereby the data is recorded before completing composition of the frame and before the first data is replaced with second data during composition of the frame.

[0013] In another aspect, a method includes copying video data representing a first video layer in multiple video layers to together establish a single video frame of a computer simulation, the single frame including at least one object, and using the video data to modify an appearance of the at least one object in the video simulation.

[0014] The details of the present application, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a block diagram of an exemplary system in accordance with the present principles; [Figure 2] 1 shows a simplified architecture. [Figure 3] 1 illustrates schematically the multiple layers of video data used to compose a single complete video frame. [Figure 4] 1 illustrates an architecture with multiple data buffers for rendering video frames. [Figure 5] In accordance with the present principles, exemplary logic is presented in exemplary flow chart form. [Figure 6] In accordance with the present principles, an exemplary buffer capture script is provided. [Figure 7] 1 shows a screenshot of a rendered frame from a legacy computer game. [Figure 8] 1 shows a screenshot of a rendered frame that has been enhanced compared to FIG. 7 to present foreground objects in three dimensions (3D) using buffer capture during rendering. [Figure 9] Using buffer capture during rendering, we show a screenshot of a rendered frame that has been enhanced compared to Figure 7 to present objects in more distinctive colors. [Figure 10] Shown is a screenshot of a rendered frame with relative enhancements compared to Figure 7 to highlight objects using buffer capture during rendering. DETAILED DESCRIPTION OF THE INVENTION

[0016] The present disclosure relates generally to computer ecosystems, including, but not limited to, aspects of consumer electronics (CE) device networks, such as computer gaming networks. The systems herein may include server and client components that may be connected over a network, whereby data may be exchanged between the client and server components. The client components may include one or more computing devices, including gaming consoles such as Sony PlayStation® or gaming consoles made by Microsoft® or Nintendo® or other manufacturers, extended reality (XR) headsets such as virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g., smart TVs, Internet-enabled televisions), portable computers such as laptops and tablet computers, and smartphones and other mobile devices, including additional examples described below. These client devices may operate in a variety of operating environments. For example, some of the client computers may use, by way of example, the Linux® operating system, the Microsoft® operating system, or the Unix® operating system, or operating systems produced by Apple® or Google®, or the Berkeley Software Distribution® or Berkeley Standard Distribution (BSD)® OS (including derivatives of BSD). These operating environments may be used to run one or more browsing programs, such as browsers made by Microsoft®, Google®, or Mozilla®, or other browser programs capable of accessing websites hosted by Internet servers as described below. Also, an operating environment according to the present principles may be used to run one or more computer game programs.

[0017] Servers and / or gateways may be used, which may include one or more processors that execute instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, clients and servers may be connected through a local intranet or virtual private network. The server or controller may be instantiated by a gaming console such as a Sony PlayStation®, personal computer, etc.

[0018] Information may be exchanged between clients and servers over a network. For this purpose and for security, the servers and / or clients may include firewalls, load balancers, temporary storage, proxies, and other network infrastructure for reliability and security. One or more servers may form an apparatus that implements a method for providing network members with a secure community, such as an online social website or a gamer network.

[0019] The processor may be a single-chip processor or a multi-chip processor capable of performing logic through various lines such as address lines, data lines, and control lines, as well as registers and shift registers. Processors including digital signal processors (DSPs) may be circuit embodiments.

[0020] Components included in one embodiment can be used in other embodiments in any suitable combination. For example, any of the various components described herein and / or shown in the figures can be combined, interchanged, or eliminated from other embodiments. "A system having at least one of A, B, and C" (and similarly "a system having at least one of A, B, or C" and "a system having at least one of A, B, and C") includes systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together.

[0021] Referring now to FIG. 1 , an exemplary system 10 is shown, which may include one or more of the exemplary devices described above and in detail below in accordance with the present principles. A first exemplary device included in system 10 is a consumer electronics (CE) device, such as an audio-video device (AVD) 12, such as a theater display system, which may be projector-based, or an Internet-enabled television with a television tuner (as well as a set-top box that controls the television). Alternatively, AVD 12 may also be a computer-controlled Internet-enabled (“smart”) phone, a tablet computer, a notebook computer, a head-mounted display (HMD) and / or headset, such as smart glasses or a VR headset, another wearable computer-controlled device, a computer-controlled Internet-enabled music player, computer-controlled Internet-enabled headphones, a computer-controlled Internet-enabled implantable device, such as an implantable skin device, etc. Regardless, it should be understood that AVD 12 is configured to implement the present principles (e.g., communicate with other CE devices to implement the present principles, execute the logic described herein, and perform any other functions and / or operations described herein).

[0022] Thus, to implement such principles, AVD 12 can be established with some or all of the components shown. For example, AVD 12 can include one or more touch-enabled displays 14, which can be implemented with high-resolution or ultra-high-resolution "4K" or higher flat screens. Touch-enabled display(s) 14 can include, for example, a capacitive or resistive touch-sensing layer with a grid of electrodes for touch sensing consistent with the present principles.

[0023] AVD 12 may also include one or more speakers 16 for outputting audio in accordance with the present principles, and at least one additional input device 18, such as an audio receiver / microphone, for inputting audible commands to AVD 12 to control it. The exemplary AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, a WAN, or a LAN, under the control of one or more processors 24. Thus, interface 20 may be, but is not limited to, a Wi-Fi® transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that processor 24 controls AVD 12 to implement the present principles, including other elements of AVD 12 described herein, such as controlling display 14 to present images on and receive input from display 14. Furthermore, it should be noted that the network interface 20 may be a wired or wireless modem or router, or any other suitable interface, such as a wireless telephony transceiver or the Wi-Fi transceiver mentioned above.

[0024] In addition to the above, AVD 12 may also include one or more input and / or output ports 26, such as, for example, a High-Definition Multimedia Interface (HDMI®) port or a Universal Serial Bus (USB) port for physically connecting to another CE device, and / or a headphone port for connecting headphones to AVD 12 to provide audio from AVD 12 to a user via headphones. For example, input port 26 may be connected wired or wirelessly to a cable or satellite source 26a of audio-video content. Thus, source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, source 26a may be a game console or disc player containing content. When implemented as a game console, source 26a may include some or all of the components described below in connection with CE device 48.

[0025] AVD 12 may further include one or more computer memory / computer-readable storage media 28, such as non-transitory, disk-based or solid-state storage, which in some cases are embodied within the AVD's chassis as a standalone device, or as a personal video recording device (PVR) or video disc player either internal or external to the AVD's chassis for playing AV programs, or as removable memory media, or as a server, as described below. In some embodiments, AVD 12 may also include a position or location receiver, such as, but not limited to, a cellular or mobile phone receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic location information from a satellite or cellular base station and provide the information to processor 24 and / or determine the altitude at which AVD 12 is located in conjunction with processor 24.

[0026] Continuing with the description of AVD 12, in some embodiments, AVD 12 may include one or more cameras 32, which may be a thermal imaging camera, a digital camera such as a webcam, an IR sensor, an event-based sensor, and / or a camera integrated into AVD 12 and controllable by processor 24 to collect pictures / images and / or video in accordance with the present principles. Also included in AVD 12 may be a Bluetooth transceiver 34 and other NFC elements 36 for communicating with other devices using Bluetooth and / or near field communication (NFC) technology, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.

[0027] Furthermore, AVD 12 may include one or more auxiliary sensors 38 that provide input to processor 24. For example, one or more of the auxiliary sensors 38 may include one or more pressure sensors that form a layer of touch-enabled display 14 itself, and may be, without limitation, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive strain gauges, optical pressure sensors, electromagnetic sensors, etc. Examples of other sensors include pressure sensors, motion sensors such as accelerometers, gyroscopes, cyclometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, gesture sensors (e.g., sensors for detecting gesture commands). Thus, sensors 38 may be implemented by one or more motion sensors such as individual accelerometers, gyroscopes, and magnetometers, and / or by an inertial measurement unit (IMU), which generally includes a combination of accelerometers, gyroscopes, and magnetometers, or by an event-based sensor such as an event detection sensor (EDS), to determine the location and orientation of AVD 12 in three dimensions. An EDS consistent with the present disclosure provides an output indicative of a change in light intensity sensed by at least one pixel of the light-sensing array. For example, if the light sensed by the pixel is decreasing, the output of the EDS may be −1, and if the light is increasing, the output of the EDS may be +1. No change in light intensity below a certain threshold may be indicated by an output binary signal of 0.

[0028] The AVD 12 may also include a wireless television broadcast port 40 for receiving over-the-air (OTA) TV broadcasts, which provides input to the processor 24. In addition to the above, it should be noted that the AVD 12 may also include an IR transmitter and / or receiver and / or transceiver 42, such as an infrared (IR) data association (IRDA) device. A battery (not shown) may be provided to power the AVD 12, which may be a kinetic energy harvester that converts kinetic energy into electrical power to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field programmable gate array 46 may also be included. One or more haptic / vibration generators 47 may be provided to generate haptic signals that can be detected by a person holding or touching the device. Thus, the haptic generator 47 may vibrate all or part of the AVD 12 using an electric motor connected to an eccentric and / or unbalanced weight via the motor's rotatable shaft, such that the shaft rotates under the control of the motor (which may be controlled by a processor such as processor 24) to produce vibrations of various frequencies and / or amplitudes, and simulation of forces in various directions.

[0029] A light source such as a projector, such as an infrared (IR) projector, may also be included.

[0030] In addition to the AVD 12, the system 10 may include one or more other CE device types. In one example, the first CE device 48 may be a computer game console that can be used to transmit computer game audio and video to the AVD 12 via commands sent directly to the AVD 12 and / or via a server, as described below, while the second CE device 50 may include similar components to the first CE device 48. In the example shown, the second CE device 50 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by a player. The HMD may include a transparent or opaque head-up display for presenting AR / MR content or VR content (or, more generally, augmented reality (XR) content), respectively. The HMD may be configured as a glasses-type display or as a large YR-type display sold by computer game console manufacturers.

[0031] In the example shown, only two CE devices are shown, and it should be understood that a fewer or greater number of devices may be used. The devices herein may implement some or all of the components shown for AVD 12. Any of the components shown in the following figures may incorporate some or all of the components shown for AVD 12.

[0032] Referring now to the at least one server 52 described above, the server 52 includes at least one server processor 54, at least one tangible computer-readable storage medium 56, such as disk-based storage or solid-state storage, and at least one network interface 58 that, under the control of the server processor 54, enables communication with other illustrated devices over the network 22, and may, indeed, facilitate communication between the server and client devices in accordance with the present principles. It should be noted that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface, such as, for example, a wireless telephony transceiver.

[0033] Thus, in some embodiments, server 52 may be an entire Internet server or server "farm" and may include or perform "cloud" functions, such that devices of system 10 may access a "cloud" environment via server 52, for example, in an exemplary embodiment relating to a network gaming application. Alternatively, server 52 may be implemented by one or more gaming consoles or other computers in the same room or nearby as the other devices shown.

[0034] The components shown in the following figures may include some or all of the components shown herein. Any user interfaces (UIs) described herein may be integrated and / or extended, and UI elements may be mixed and matched between UIs.

[0035] The present principles may use a variety of machine learning models, including deep learning models. Machine learning models consistent with the present principles may use a variety of algorithms trained using methods including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms that may be implemented by computer circuitry include one or more neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and a type of RNN known as a long short-term memory (LSTM) network. Support vector machines (SVMs) and Bayesian networks may also be considered examples of machine learning models. In addition to the types of networks listed above, the models herein may be implemented by classifiers.

[0036] As understood herein, performing machine learning may therefore involve accessing a model against training data and then training the model to enable the model to process additional data to make inferences. Thus, an artificial neural network / artificial intelligence trained by machine learning may include an input layer, an output layer, and multiple hidden layers in between, which are configured and weighted to infer about an appropriate output.

[0037] Referring now to FIG. 2, a computer simulation engine, such as a legacy computer game engine 200, issues commands to one or more GPUs 202 to render requested simulation video frame by frame. While the operating system 204 on which the game engine runs can detect commands as they are sent to the GPU, the legacy game engine itself may not communicate its rendering scheme to the OS 204 separately from the commands to the GPU. Thus, the OS 204 may access information programs, such as scripts, that are built to understand the rendering scheme and inform the OS when and where to capture data during the rendering of one or more frames. By "capture," we mean recording or copying data representing frame elements that the script author can use to enhance one or more objects in the simulation to make them more visible to others or to enhance their appearance in other ways described herein.

[0038] 3 shows multiple frame layers 302 (four frame layers are shown for illustrative purposes only) rendered in succession. One frame layer may represent depth information, one may represent color information, one or more layers may represent lighting information, one or more layers may represent transparency information, one or more layers may be used for opaque geometry information (such as spraying), etc.

[0039] Thus, in some cases, information may be spread across several layers. For example, some of the final composited frame may be made up of parts from different layers. The script described elsewhere in this document may specify that explicitly. Ultimately, the script is trying to backtrack what was presented on the screen from what it originated from.

[0040] Additionally, note that portions of layers can be moved relative to each other, so a script can specify that one particular layer be presented on one portion of the screen and another layer be presented on another portion of the screen. As an example, if heads-up display (HUD) elements are rendered separately in small buffers, they may only be displayed in the corners of the screen.

[0041] As understood herein, with such layer-by-layer rendering, important information for image generation may be lost by the time the complete image frame 300 is fully constructed. To avoid this, as described herein, a customized capture script or other tool may be created for a computer game that specifies what data to capture for future use during the rendering of a game video frame, such as frame buffer data, depth buffer data, and stencil buffer data. The script also specifies when during frame rendering to record the data. By capturing data mid-frame, the boundaries of objects on the screen and the relative distances between objects can be inferred. This information can be used for 2D to 3D conversion, coloring objects for increased contrast, highlighting objects, etc.

[0042] The problem addressed here is the ability to use the same buffer multiple times when rendering a single frame with multiple layers. For example, a color buffer may be used to render objects in color in one layer, but then the data may be flushed and reused to render a lighting layer. When examining a color buffer after frame compositing, it is not always clear what data is actually being examined unless the present principles are applied. The same applies to buffers nominally referred to as "depth" buffers, which may not have any depth-related data in them at the completion of frame compositing.

[0043] 3 essentially describes a particular example configured as a forward renderer. It should be understood that the present principles also apply to deferred rendering. In such an embodiment, various layers may include "buffers" with material information, depth, and other data.

[0044] 4 provides additional details. In rendering video, GPU 202 may access data buffers, including, by way of example, color buffer 400, depth buffer 402, and stencil buffer 404. Data from the buffers is written to rendering engine 406 for presentation to display 408, such as any of the displays described herein.

[0045] 5 shows an example of the overall logic. Starting at block 500, it is determined how a particular legacy game engine structures its frames in terms of which layers it uses for each frame and the buffers used for each layer. This determination may be made by accessing the game's software or by observing in a lab environment the game commands issued to the GPU and then identifying what the commands are.

[0046] Proceeding to block 502, it is determined what data is of interest for frame rendering. This data may include, for example, depth data, or color data, or other data useful for the designer's intended purpose. Once this information is compiled and, for example, reflected in a script or other format, it is provided to OS 204 of FIG. 2, allowing the OS to capture the desired data from the appropriate format during the rendering of one or more frames via a "buffer grab" (copying or recording the data into a buffer before the buffer is flushed for subsequent use in rendering another layer of the frame).

[0047] Proceeding to block 506, the data captured in block 504 is used to enhance the image, for example, by modifying the appearance of at least one object in the computer simulation. This may be for accessibility reasons, for example, by changing the color of the object to a different color so that it is more easily identified by some people, or by highlighting the object to draw attention to it. Or, an object rendered in 2D by the legacy game may be rendered in 3D in a remastered version of the game. These modifications may be made dynamically, i.e., on subsequent frames of the legacy game as presented, or offline in a remastered version of the game; the modifications are typically made at the OS level rather than at the game level.

[0048] In another embodiment, a highlighted version of the frame may be presented to a viewer of the game or for replay of a particular scene in a newly introduced "replay mode" or highlight reel, for example, highlighting which opponent shot a player in a gunfight, or highlighting a football player who was about to sack the quarterback in a sports game.

[0049] Note that in addition to the above, data can be captured from memory as described by the script. Such additional data can be prepared by the CPU.

[0050] Figure 6 shows an exemplary script 600 that may receive as input commands (simple examples are the first, second, and third draw calls) from the game engine 200 of Figure 2 to the GPU 202. The script knows whether the current layer being drawn is an opacity layer, a depth layer, a color layer, a lighting layer, etc., and which buffer the data for that layer is in, so that the desired data (such as color data or depth data) can be captured when the buffer in which the data is located is valid for the desired data.

[0051] Thus, the illustrated non-limiting example script 600 may include lines 602, each of which may correspond to a single layer of multiple layers in a single frame. The name of the layer may also be included. A command 604 may be displayed for at least one of the layers to capture data from a particular buffer, such as a color buffer or a depth buffer, before rendering the next layer. Once the final layer is rendered, an indicator 606 may be provided after the final layer to indicate that compositing of the frame is complete.

[0052] 2 may examine command codes passed from the game engine 200 to the GPU 202 to determine whether the command pertains to an object and when such a command stops, ensuring that the command pertains to the next layer being drawn. As understood herein, this is advantageous because the number of layers used to compose a complete frame may change during the game, such as when a light is emulated to be destroyed and therefore the associated lighting layer is no longer drawn in the rendered frame.

[0053] Additionally, scripting can facilitate diagonal frustrum culling, which refers to situations where depth values ​​may be accurate for object separation but may not necessarily correspond to accurate depths for 3D reconstruction. Scripts can describe how to generate accurate depth values ​​from them.

[0054] To do this, the projection matrix may be modified. More generally, some techniques, such as diagonal frustrum culling, modify the depth values ​​stored in the depth buffer, and the script provides a way to restore the depth values ​​for 3D reconstruction.

[0055] 7 shows a screenshot 700 of a rendered frame from a legacy computer game that can be presented on any display described herein, for example. In the non-limiting example shown, all objects 702 are commanded for 2D presentation by game engine 200. Specifically, a foreground object 704 may be presented in 2D and a first color and may overlap another object as shown.

[0056] Consider now Fig. 8, which shows a screenshot of a rendered frame that has been enhanced compared to Fig. 7 to present a foreground object 704 in three dimensions (3D) using an in-render buffer capture from a depth buffer. The data shows that the foreground object 704 is in front of other objects 702.

[0057] Further, consider Figure 9, which shows a screenshot of a rendered frame that has been enhanced compared to Figure 7 to present an object (in the example shown, foreground object 704) in more distinctive color using buffer capture during rendering, in this case by capturing the color buffer while it is still valid for color before it is overwritten with data used to render subsequent layers of the frame.

[0058] Consider again Fig. 10, which shows a screenshot of a rendered frame that has been enhanced compared to Fig. 7 to highlight an object (in the example shown, foreground object 704) as shown at 1000 using a buffer capture during rendering. In this case, this is done by capturing the color buffer while it is still valid for color before it is overwritten with data that will be used to render subsequent layers of the frame.

[0059] Although particular embodiments have been shown and described in detail herein, it should be understood that the subject matter encompassed by this invention is limited only by the claims.

Claims

1. 1. A device comprising at least one computer storage device containing instructions executable by at least one processor, said instructions being not transitory signals, said instructions comprising: (a) at least partially constructing a first video frame of a computer simulation, Rendering a first video layer using a first data buffer; Rendering a second video layer using the first data buffer; and capturing data in the first data buffer during rendering before completing composition of the first video frame; (b) using the data captured from the first data buffer to modify the appearance of at least one object associated with the computer simulation; The device.

2. the at least one object is rendered in two dimensions (2D), the first data buffer comprises a depth data buffer, and the instructions are executable to at least partially modify the appearance of the at least one object; The device of claim 1 , wherein the modifying is performed by rendering the at least one object in 3D in at least one frame following the first frame and / or in the first frame.

3. the at least one object is rendered in two dimensions (2D), the first data buffer comprises a depth data buffer, and the instructions are executable to at least partially modify the appearance of the at least one object; The device of claim 1 , wherein the modifying is performed by rendering the at least one object in 3D in at least one frame of a remastered version of the computer simulation.

4. the at least one object is rendered in a first color in the first frame, the first data buffer includes a color data buffer, and the instructions are executable to at least partially modify the appearance of the at least one object; 2. The device of claim 1, wherein the modifying is performed by presenting the object in a second color different from the first color in at least one frame following the first frame and / or in the first frame.

5. the at least one object is rendered in a first color in the first frame, the first data buffer includes a color data buffer, and the instructions are executable to at least partially modify the appearance of the at least one object; 10. The device of claim 1, wherein the modifying is performed by presenting the object in a second color different from the first color in at least one frame of the remastered version of the computer simulation.

6. the instructions are executable to at least partially modify the appearance of the at least one object; The device of claim 1 , wherein the modifying is performed by highlighting the object in the first frame and / or in at least one frame following the first frame.

7. the instructions are executable to at least partially modify the appearance of the at least one object; The device of claim 1 , wherein the modifying is performed by highlighting the object in at least one frame of a remastered version of the computer simulation.

8. 10. The device of claim 1, wherein the instructions are executable to use a script to identify at least one of the first frame and the first data buffer for use in capturing data.

9. 10. The device of claim 1, wherein the instructions are executable to use a script to identify both the first frame and the first data buffer for use in capturing data.

10. The device of claim 1 comprising the at least one processor.

11. at least one processor programmed with instructions, the instructions comprising: Rendering at least one frame of the computer-simulated video; During rendering, recording first data from at least one buffer used in the rendering of the frame, the data being recorded before completing compositing of the frame and before the first data is replaced with second data during compositing of the frame; A device that performs the above.

12. 12. The apparatus of claim 11, wherein the apparatus comprises at least one computer programmed with instructions to use the first data to modify an appearance of at least one object associated with the computer simulation.

13. the at least one object is rendered in two dimensions (2D), the first data buffer comprises a depth data buffer, and the instructions on the computer are executable to at least partially modify the appearance of the at least one object; The apparatus of claim 12 , wherein the modifying is performed by rendering the at least one object in 3D in the first frame and / or in at least one frame following the first frame.

14. the at least one object is rendered in two dimensions (2D), the first data buffer comprises a depth data buffer, and the instructions on the computer are executable to at least partially modify the appearance of the at least one object; The apparatus of claim 12 , wherein the modifying is performed by rendering the at least one object in 3D in at least one frame of a remastered version of the computer simulation.

15. the at least one object is rendered in a first color in the first frame, the first data buffer includes a color data buffer, and the instructions on the computer are executable to at least partially modify the appearance of the at least one object; 13. The apparatus of claim 12, wherein the modifying is performed by presenting the object in a second color different from the first color in at least one frame following the first frame and / or in the first frame.

16. the at least one object is rendered in a first color in the first frame, the first data buffer includes a color data buffer, and the instructions on the computer are executable to at least partially modify the appearance of the at least one object; 13. The apparatus of claim 12, wherein said modifying is effected by presenting said object in a second color different from said first color in at least one frame of a remastered version of said computer simulation.

17. the instructions on the computer are executable to at least partially change the appearance of the at least one object; The apparatus of claim 12 , wherein the modifying is performed by highlighting the object in the first frame and / or in at least one frame following the first frame.

18. the instructions on the computer are executable to at least partially change the appearance of the at least one object; The apparatus of claim 12 , wherein the modifying is performed by highlighting the object in at least one frame of a remastered version of the computer simulation.

19. copying video data representing a first video layer among a plurality of video layers to together establish a single video frame of a computer simulation, the single frame including at least one object; using the video data to modify the appearance of the at least one object in the video simulation; and A method comprising:

20. 20. The method of claim 19, wherein the video data is copied from a buffer.

Images