Application process context compression and replay

Abstract

The application state data from the main memory may be compressed, and the compressed data may be written to a first location of the mass storage. Updated application state data is generated, and the updated application state data is compressed from the main memory. The updated application state data is then written to a second location of the mass storage. Processing may then be paused on the application state data and the updated application state data. The compressed application state data and the compressed updated application state data stored in the mass storage are scanned, and information corresponding to the compressed application state data and the updated compressed application state data stored in the mass storage is displayed using information from the scanned compressed application state data and the compressed updated application state data.

Description

[Technical Field]

[0001] Aspects of the present disclosure relate to memory management, and in particular to application replay from compressed main memory pages. [Background technology]

[0002] Modern computer architectures use main memory to store data manipulated by the processor. The processor can transfer information from main memory to its registers, perform operations, and then transfer the register contents back to main memory. Such main memory contains application state data, which is a snapshot of the state of data being processed by the processor that has been moved from the registers to main memory. Application state data can include application context such as processor registers, stacks, and heaps. Main memory data is periodically saved to non-volatile memory, also known as mass storage, such as a hard disk drive (HDD) or solid-state drive (SSD). During processing, the processor can update application state data in main memory without updating data stored in mass storage; this is sometimes referred to as a dirty page, because the page data is updated without being written to mass storage. Application state data is stored in main memory in one or more pages, each consisting of multiple fixed-size page frames or variable-size page segments. The processor maintains an address buffer that holds the location of application state data in main memory.

[0003] Many operating systems implement memory swapping, which allows the operating system to increase the size of main memory without requiring additional physical memory. Memory swapping uses a portion of nonvolatile memory storage space as main memory. Swapping retrieves application state data from main memory and writes it to a nonvolatile memory space while the application processing is paused. The operating system may swap application state data written to this area of ​​nonvolatile memory back and forth between the physical memory of main memory and the nonvolatile memory area as the operating system changes the application being processed. The swapping of pages from physical memory to nonvolatile storage and the restoration of pages from nonvolatile memory to physical memory are controlled by two different algorithms. When an application attempts to allocate a page of memory but physical memory is full, the operating system pauses the application and selects a page that is not actively in use to copy to nonvolatile memory. The operating system then reuses that page for memory allocation and resumes the application. Another reason for swapping out pages is application context switching. When an application is switched out, its memory page is also swapped out to make room for the application being swapped in.

[0004] It has been proposed that improved memory management efficiency can be achieved by compressing application data stored in portions of main memory. Various compression techniques have been proposed to reduce the size of application state data stored in main memory. These techniques have not been adopted industry-wide.

[0005] Video games and other applications save application state data using separate points programmed within the application. These are often referred to as "save games," "save states," or "save files." These save files may be updated or created periodically by the game application or at the user's discretion, depending on the implementation. Save data creation operates independently of processor memory management, creating save data in main memory or mass storage independent of memory swapping. The contents of save data are variable depending on the implementation and are not standardized across applications.

[0006] It is in this context that aspects of the present disclosure arise. Summary of the Invention

[0007] The teachings of the present disclosure can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which: [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 10 is a timing diagram illustrating application state compression according to aspects of the present disclosure. [Figure 2] FIG. 1 is a flow diagram illustrating application state compression in a method for application state compression and replay according to an aspect of the present disclosure. [Figure 3] FIG. 1 is a flow diagram illustrating application state selection in a method for application state compression and replay according to an aspect of the present disclosure. [Figure 4] FIG. 10 is a flow diagram illustrating application state replay loading in a method for application state compression and replay according to an aspect of the present disclosure. [Figure 5] FIG. 2 is a block diagram illustrating the contents of a memory page containing application state, according to aspects of the present disclosure. [Figure 6]FIG. 1 is a flow diagram illustrating application state compression in a method for application state compression and replay according to an aspect of the present disclosure. [Figure 7] FIG. 10 is a flow diagram illustrating updated application state compression in a method for application state compression and replay according to an aspect of the present disclosure. [Figure 8] FIG. 10 is a timing diagram illustrating application state compression in a method for application state compression and replay according to an aspect of the present disclosure. [Figure 9] FIG. 10 is a timing diagram illustrating application state loading and replay in a method for application state compression and replay according to an aspect of the disclosure. [Figure 10] FIG. 1 is a block schematic diagram of a system implementing a method for application state compression and replay according to an aspect of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] Although the following detailed description contains many specific details for purposes of illustration, those skilled in the art will recognize that many variations and modifications to the following details are within the scope of the invention. Accordingly, the exemplary embodiments of the invention described below are set forth without loss of generality to, and without imposing limitations on, the claimed invention.

[0010] Although it has been shown that page compression can be used to improve memory management efficiency, these techniques have not been widely adopted due to a lack of hardware resources. Fast multi-core processors are now commonplace, and hardware resources are abundant enough to implement page compression without impacting processing. Memory management systems and other applications have not yet taken advantage of memory compression.

[0011] While previously proposed memory compression methods involve directly compressing memory pages into main memory during processing, aspects of the present disclosure provide increased system functionality by compressing and retaining compressed memory pages in mass storage. Retaining compressed memory pages in mass storage enables application replay, which allows a user to revert the processing of an application to a previous point during processing. For example, but not by way of limitation, retaining compressed memory pages in mass storage may enable playback of a video game from a previous point in a video game where the game was not saved, restarting a simulator program at a point between simulations where there was no state saving, undoing changes made in a drawing or writing program where the previous state was not saved, reviewing the retained compressed application state during program debugging to determine the application's context and discover bugs, and using the retained compressed application state to recover application data lost after a system crash. The benefits of retaining compressed application state are numerous and allow for the addition of functionality to programs that previously lacked the ability to save state or that did not have frequent save points. Additionally, the compressed application state may contain more information than the application-created save files, allowing for more granular application replay or more user-friendly application replay points.

[0012] method FIG. 1 illustrates an implementation of application state compression according to aspects of the present disclosure. In this particular implementation, an application, e.g., a gaming application, executes processing over predetermined time intervals and generates output in the form of data corresponding to image frames at the end of each interval. However, aspects of the present disclosure are not limited to such implementations. As shown, initially, a system may execute processing and generate a first image frame 101. After generating the first image frame, the system pauses at 102, and application state data paged in main memory is copied at 103 to a virtualized main memory space in a first location of mass storage. The application state data in the main memory pages is then compressed at 104 and copied to a second location of mass storage at 105. As used herein, and as commonly understood by those skilled in the art, “state” is a general term that refers to all information necessary to define a system. The “context” of an application more specifically refers to all information necessary to restore a computer application.

[0013] The processor resumes processing the application state data and a second image frame 106 is generated. Generating the second image frame 106 may include generating updated application state data that is stored in main memory. In some implementations, the application process may be paused at 107 and the updated application state data generated in the second frame is copied from the physical main memory to the virtual main memory in a first location of mass storage at 108. The application state data generated in the second frame may then be compressed at 109 and copied to a third location of mass storage at 110.

[0014] In an alternative embodiment, one or more image frames may be processed with the corresponding updated application state data or application data without compressing the memory pages of the updated application state data. Accordingly, elements 107-110 are shown with dashed lines indicating that the operations are optional depending on the embodiment. In another alternative embodiment, the application state data copied at 103 and the updated application state data copied at 108 may also be stored in physical memory. The copied pages corresponding to the state data are a snapshot of the application. Memory compression may be performed on the snapshot of physical memory.

[0015] As shown, a third image frame 111 may be generated with updated application state data. Processing may then be paused at 112, and memory pages corresponding to the updated application state data may be copied to the virtual main memory space at a first location in mass storage at 113. The updated application state data generated for the third image frame may then be compressed at 114 and copied to a fourth location in mass storage. Alternatively, if the previous updated application state data was not compressed, the compressed updated application state data may be copied to a third location in mass storage. This process may continue up to an nth image 115, where n is the number of image frames generated while executing the application.

[0016] 1, in some implementations, the pause in processing may only last long enough to copy application state data from main memory to virtual main memory space. According to some aspects of the present disclosure, parallel processing with a processor (e.g., a second thread, core, or processing unit), a GPU, a discrete codec, etc. may be used to compress and copy the application state data to mass storage outside of the swap operation. In some implementations, a processor other than the CPU, for example, a GPU, may be used to decompress the compressed application state data or compressed-updated application state data.

[0017] Furthermore, it should be noted that in some cases, processed application state data may be saved to mass storage independently of the compression paging method of the present disclosure. For example, as described above, an operating system may swap some inactive pages from physical memory. If these pages are “clean,” i.e., unchanged from the state of the previous interval or frame, they can be directly compressed and stored without unpacking the pages to physical memory. However, if the pages are “dirty,” i.e., changed from the state of the previous interval or frame, the latest memory contents from physical memory must be selected and stored. In such situations, “clean page” and “dirty page” flags may be used as indicators to assist the memory compressor. If a page can remain clean after one processing interval, e.g., one frame iteration, the compressor need only remove this page from the current frame context and use an index to reference the page to the context of the previous frame. Otherwise, the stored dirty page is stored with the context for the current interval, such as the current frame. When using the “dirty page” flag, it is desirable to synchronize page memory swapping with the operation of the application state compressor.

[0018] If the application state compressor is running in conjunction with the virtual memory management, memory page swapping may be performed synchronously with application state compression. In other words, the operating system only performs page memory swapping when the application compressor needs that page memory for compression. In this case, the dirty page flag signifies that the page is being modified between application state data and updated application state data. Alternatively, if the operating system swaps memory pages independently of the application state compressor, dirty pages are only pages that have been modified since the last page swap, but the timing of the last swap is independent of application state data compression and updated application state compression. In this situation, the dirty page flag is not useful for determining whether the data in the flagged page has been modified, and the application compressor should always assume that the data will be modified.

[0019] As noted above, aspects of the present disclosure are not limited to only the implementation shown in FIG. 1. FIG. 2 more generally illustrates details of application state compression according to aspects of the present disclosure. A method for application state compression and replay can begin 201 with the selection of an application. A processor may pull application data from mass storage to main memory for processing. The processor may then process the application and generate application state data. After pausing processing, the application state data is copied from processor registers to pages of main memory, as shown at 202. In some optional implementations, the processor may perform a swap operation by copying the application state data from physical main memory to a first location, such as a virtual memory space in mass storage or to a main memory location at 203. Next, the processor reads and compresses the application state data from the first location, as shown at 204. In some alternative implementations, the swap operation can be skipped, and instead, the processor can read and compress the application state data directly from main memory at 204, such that no swap operation is required for application state compression. The compressed pages of application state data are then written to a second location in mass storage, as shown at 205. Because this second location in mass storage is outside of the virtualized main memory space, the processor does not operate on this area of ​​mass storage as if it were main memory. After processing is paused, the processor may resume processing the application state data, processing one or more image frames from the application state data to create updated application state data that is stored as pages in physical main memory at 206.

[0020] As before, the processor is paused and the updated application state data may be (optionally) swapped from the pages of physical main memory into virtual main memory space in a first location in mass storage at 207. The updated application data is then compressed at 208 and written to a third location in mass storage at 209. Note that compression of the updated application state data need not be performed at each processing interval, e.g., after a full image frame, but may occur at intervals such as every two frames, every three frames, every five frames, or every n frames, where n is determined by the application, the operating system, or based on available resources. Alternatively, n may be user-determined.

[0021] The processor can continue to process the application state data and generate image frames and updated application state data at 210, or the application can finish processing and compressing the application state as shown at 211. A termination state can occur when a processing end state or program interruption is reached; alternatively, a termination state can be a user-selected application processing pause to return to a previous application state or updated application state.

[0022] In many use cases, it is desirable to synchronize the swapping out of pages with application state compaction: instead of swapping out pages or switching applications when physical memory is full, pages can be periodically copied from one location to another for application state compaction.

[0023] Additionally, because swapped memory pages are compressed before being stored, standard operating system processes can be modified to store swapped pages elsewhere in physical memory instead of non-volatile memory. Typically, data compressors operate more easily in physical memory; only the compressed data is stored in non-volatile memory.

[0024] As mentioned above, memory page swapping can be performed synchronously with application state compaction or independently of application state compaction, and clean and dirty page flags, as described above, can be used in conjunction with such implementations.

[0025] FIG. 3 is a block diagram illustrating application state selection in a method for application state compression and replay according to aspects of the present disclosure. Initially, the processor may enter the application state selection method from the application state compression method due to an interruption in application processing, such as a programmed interruption or application failure, or due to user input. The processor pauses processing on the current application state data, as shown at 301. Next, the system may scan mass storage for compressed and updated application state data at 302. In some optional implementations, the system may reconstruct image frames associated with one or more states from the application state data or updated application state data 303. Furthermore, in some other alternative implementations, audio associated with one or more states from the application state data may be reconstructed from an audio buffer of the application state data or updated application state data 303. The audio may be played, for example, but not limited to, through a speaker, and may be played with or without the display of image frames. According to aspects of the present disclosure, some implementations of the application state data and updated application state data may include associated image frames and / or audio buffers that are compressed using the application state data or the updated application state data. Decompression of the compressed image frames and / or audio buffers in the application state data or the updated application state data may reconstruct the image frames for display or the audio for playback. Alternatively, the application context may be processed to reconstruct the image frames or audio buffers.Using information from the scanned compressed application state data and compressed updated application state data, the system can display or play information corresponding to the compressed application state data and compressed updated application state data in mass storage at 304. In such optional implementations, this information may include reconstructed image frames or audio buffers generated from the compressed application state data or compressed updated application state data.

[0026] In some embodiments, a list of the compressed application state data and the compressed updated application state data may be displayed, and the list may be arranged in chronological order. To facilitate rapid chronological organization of the display, each instance of the compressed application state data and the compressed updated application state data may include tags describing attributes such as the date and time of creation and the instance of the application. A display of the information corresponding to the compressed application state data and the compressed updated application state data may be generated, for example, without limitation, by displaying the information in the tags on a display screen. In some embodiments, this may also include displaying reconstructed image frames and / or playing audio buffers reconstructed from the compressed data. Once the information corresponding to the compressed application state data and the compressed updated application state data is displayed, the processor 305 waits for a user selection. In some alternative embodiments, image frame data from the compressed application state data or the compressed updated application state data may be displayed upon selection by the user 306. In some embodiments, audio buffer data from the compressed application state data or the compressed updated application state data may be played upon selection by the user 306, with or without the display of image frames. This embodiment can reduce wasted decompression cycles and provide an intuitive idea for representing the selected state of the application while still providing functionality to the user. The decompressed image frame and / or audio buffer can be rebuilt before the user confirms that it is desired to resume the application from the selected compressed or compressed-updated application state. Assuming that the compressed or compressed-updated application state is selected or confirmed as selected by the user, the system can start the application at 307 using the compressed or compressed-updated application state data, for example, as described below with respect to FIG. 4.

[0027] The flow diagram shown in FIG. 4 illustrates application state replay loading in a method for application state compression and replay according to an aspect of the present disclosure. After selecting a compressed application state or a compressed-updated application state at 401, the system initializes processing from the compressed application state or the compressed-updated application state. To begin, the compressed application state or the compressed-updated application state is decompressed as shown at 402. In some implementations, the compressed application state data or the compressed-updated application state data is copied to memory, where a processor performs operations to decompress the data. In alternative implementations, a separate processing unit, such as a graphics processor, or a discrete compression / decompression circuit, such as a video encoder / decoder (CODEC), may perform the decompression and copy the compressed application state data or the compressed-updated application state data to reserved memory space for the separate processing unit. To decompress the compressed application state data or updated application state data, the system can use the inverse of the algorithm used to compress the data to decompress the data, which may include using the same dictionary used for dictionary encoding using the inverse operation of a common compression algorithm or applying X-ORd data to a base data set of inter-predicted data. Before, during, or after decompression, in some optional implementations, the application state data in main memory where processing is paused can be cleared at 403 to make room for the decompressed application state data or the decompressed updated application state data. Alternatively, the portion of the application state data in main memory where processing is paused can be retained, and one or more pointers to the corresponding memory location(s) can be changed to the location(s) of the decompressed application state data or the decompressed updated application state data when copying it to main memory.Once decompressed, the compressed application state data or the compressed-updated application state data may be copied to main memory at 404. As described, the decompressed-compressed application state data or the decompressed-updated application state data may be copied to the same location as the application state data or the updated application state data that was previously in process. Alternatively, as discussed above, the decompressed-compressed application state data or the decompressed-updated application state data may be copied to a new location in main memory, and the pointer(s) of different tables (such as the translation lookaside buffer) may be changed. Processing of the application may be resumed in the same manner as a normal context switch, e.g., from one application to another. Specifically, a normal context switch process may include a process of loading registers from memory to the processor before starting the processor. Such a loading process, sometimes referred to as initialization, may be performed before processing is resumed. Finally, after the decompressed-compressed application state data or the decompressed-updated application state data is copied to main memory and any necessary initialization is performed, the processor may be unpaused at 405 and begin processing the state data copied in main memory.

[0028] Thus, in practice, according to some aspects of the present disclosure, a gamer can use the captured gameplay video (reconstructed from the compressed application state data and the compressed updated application state data) to search for replay points. The game video may be played forward or backward. When the gamer finds the appropriate replay point, the gamer may pause the video and start replay at that point. To enable such use cases, a series of time codes may be embedded as tags in both the compressed application state data and the compressed updated application state data. When the gamer pauses the video at a particular frame, the time code of the video frame may be passed to decompression hardware and / or software to obtain the application state or updated application state at that time code. Note that the frame rate of the game video may differ from the rate at which the application context is captured. There may not be a one-to-one match of time codes between the captured context and the captured video.

[0029] compression The block diagram shown in FIG. 5 illustrates an example of the contents of a memory page containing application state, according to an aspect of the present disclosure. As shown, memory page 501 is composed of multiple fixed-size frames 502. In some alternative implementations, the memory page may be composed of multiple variable-sized segments, and the size of each segment may be recorded in a table. Application state data and updated application state data 507 may span multiple frames or segments of memory page 501. In some implementations, application state data and updated application state data may include both image frame data and / or audio buffer data 503 and associated application context data 504. Alternatively, application state data or updated application state data may be application context data without an associated image frame or audio buffer. Note that an image frame is data representing pixel values ​​for display on a display screen. An audio buffer is data representing samples of an audio signal for playback on a speaker or other sound-generating device. Application context data may include application data, additional main processor and GPU registers, GPU memory and local memory of GPU compute units, a stack, and a heap. The stack is a linear data structure that tracks variables used by the processor during processing in an ordered manner. The heap is a large unordered memory space for other variables and memory objects created during processing. Memory pages may also contain other data, such as data for other applications or operating system processes 506. Although memory pages 501 are shown separated into frames, memory pages may be partitioned into smaller memory blocks. Larger compressed memory block sizes may improve compression efficiency, while smaller block sizes may result in faster compression. The compressor may determine an appropriate compression model for each block. Statistically, larger block sizes result in better compression.In some implementations, compressed memory blocks may be variably sized using tags or tables that provide characteristics of the memory blocks, such as, but not limited to, size and location. In some implementations, the compressed application state data after compression and the compressed updated application state data may include decompressed tags that describe attributes of the data, such as creation time, application, and size. The tags may also indicate which portions of the compressed data are image frames or audio buffers. Tags may be compressed or uncompressed. Uncompressed tags provide faster retrieval speeds. Compressed tags provide better compression efficiency.

[0030] The flow diagram shown in FIG. 6 illustrates an example of application state compression in a method for application state compression and replay according to an aspect of the present disclosure. First, the application state data may be modeled to find available regularities within the data in a given page, block, segment, or other data unit, as shown in 601. The modeling may determine regularities such as the number of zeros in the data or the repetition of specific data strings or structures. After modeling, the device can determine the best compression algorithm by applying different compression algorithms 605, 606, and 603 and measuring the compression of the resulting compressed application state data 602. Compression may be measured using the final memory size of the compressed application state data. The memory sizes of the compressed application state data created by general compression 605, lexicographic compression 606, and run-length compression 603 may be compared in 604 to determine the compressed application state data with the smallest memory size. The compressed application state data with the smallest size may then be stored in mass storage 607. In some alternative implementations, the compression ratio may be obtained using the final compressed file size. The compression ratio may be the ratio between the memory size of the compressed application state data and the memory size of the uncompressed application state data. Compression ratios may be generated for a general compression algorithm 605, a dictionary compression algorithm 606, and run-length encoding 603. All ratios of the applied compression algorithms may be compared in 604, and the algorithm resulting in the smallest memory size may be selected and stored in mass storage 607.

[0031] If modeling and initial compression determine that a unit of application state data in question contains many repeated zero or single digits, then run-length encoding can be applied to that unit of data stored in mass storage. Run-length encoding takes each repeated single digit and reduces it from many repeated single digits to a two-entry encoding that presents the digit and the number of times it is repeated, in order. For example, and not by way of limitation, a sequence such as AAAAABBBBCCDDDDD encoded with run-length encoding is A5B4C2D5, thus reducing a 16-character sequence to 8.

[0032] The modeling data and initial compression may determine whether there are any available regularities in the data based on the size or compression ratio of the compressed application state data. If the application state data contains available regularities, then dictionary coding, such as Lemple-Ziv-Welsh (LZW) compression, DEFLATE compression, or Lemple-Ziv-Huffman-Arithmetic-Markov (LZHAM) compression, may be applied to the compressed application state data stored in mass storage. Dictionary coding may compress the application state data by replacing regularities in the application state data with tokens and arranging the tokenized regularities in a table (dictionary). If there are insufficient regularities in the data to encode using dictionary compression, as determined by modeling 601 and a comparison of the compressed memory size 604 from the initial application of the compression algorithm, then the final compressed application state data stored in mass storage 607 may be compressed with another common compression algorithm, such as, but not limited to, context tree weighted (CTW) coding or other similar lossless compression algorithms.

[0033] FIG. 7 is a block diagram illustrating updated application state compression in a method for application state compression and replay according to an aspect of the present disclosure. First, memory pages of updated application state data are modeled to determine regularities and similarities between the updated application state data and previously generated application state data, as shown at 701. After modeling, the best compression algorithm can be determined by applying compression algorithms such as inter-prediction with general compression 708, inter-prediction with lexicographic compression 707, inter-prediction with run-length encoding 705, and novel compression algorithm 703 as seen in FIG. 6. Compression can be measured using the final memory size of the compressed application state data 702 generated by each algorithm 708, 707, 705, and 703. The memory sizes of the compressed application state data created by each algorithm 708, 707, 705, and 703 can be compared at 704 to determine the compressed application state data with the smallest memory size. The compressed application state data with the smallest memory size can then be stored to mass storage at 709. In some alternative embodiments, the compression ratio may be obtained using the final compressed file size. The compression ratio may be the ratio of the memory size of the compressed application state data to the memory size of the uncompressed application state data. A compression ratio may be generated for each of the compression types 708, 707, 705, and 703. All ratios of the applied compression algorithms may be compared in 704, and the algorithm resulting in the smallest memory size may be selected and stored in the mass store 709. The device may use data modeling and the memory size of the compressed state data to determine whether there is any similarity in the updated application state data with the previous application state data. If there is no similarity, the device may compress the updated application state data using a new compression algorithm, i.e., an algorithm without mutual prediction, as described above and shown in FIG. 6.If there is a similarity between the updated application state data and the previous application state data, as determined by modeling 701 and applying reciprocal prediction and compression, the device uses data modeling and compressed data size to determine that there are many repeated zero or single digits in the data. If many repeated single digits or zeros are found, reciprocal prediction with run-length encoding may be applied to the updated application state data. In reciprocal prediction, the updated application state data is exclusive-ORed (XORed) with the previous application state data to create a page of data containing only the differences between the updated and previous application state data. Run-length encoding may be applied to the page of changed data created by the XOR operation and stored in the mass store 709 based on a comparison of compressed memory sizes at 704. The run-length encoding discussed above then removes repeated single digit strings containing two entry coders. If there are insufficient repeated single digits, the device may determine, through application of a compression algorithm and comparison at 704, that there is a regularity in the data and that lexicographic compression may be used for the stored compressed state data. If there is a sufficient number of regularities in the updated application state data, as determined by applying a compression algorithm and comparing 704, the stored updated application state data may be compressed, for example with LZW, using inter-prediction with dictionary coding. When paired with inter-prediction, dictionary coding can use the same coding tables (dictionaries) as used with the previous application state data.

[0034] If there are not enough regularities in the updated application state data, cross-prediction with a general encoding method may be applied to the stored compressed state data. As described above, the general encoding method may be, for example, but not limited to, a CTW lossless compression method. In some implementations, there may be a large amount of updated application state data with few "clean pages" of application state data. Therefore, to enable random access, independent updated application state data may be inserted and compressed as application state data (shown in FIG. 6) at intervals based on the speed of decompression. For example, without limitation, if 50 compressed-updated application states can be decompressed per second, compressed independent updated application state data may be created every 25 compressed-updated application states. In this example implementation, a maximum response time of 0.5 seconds results. If the selected updated application state data is close to the independently updated application state data, the response time may be faster. The independent updated application state data includes all memory pages, not just dirty memory pages. Furthermore, if the independent updated application state data is encoded according to the method shown in FIG. 6, cross-prediction is not applied to the independently updated application state data. Inserting independent updated application state data may increase the memory footprint of the compressed data, but may require fast access times because the decompressor must start from the independent application state data and decode all application state that was updated between the independent context and the target context.

[0035] Furthermore, inter-prediction can use non-reference interdependent compressed application state data to speed up decompression. Adding non-reference interdependent updated application state data can increase the interval between independently updated application state data and / or reduce response latency. Using non-reference interdependent updated application state data in the future prevents contexts from sharing dictionaries. Pages of non-reference interdependent updated application state data cannot be used as references for future inter-predicted updated application state data. When the decompressor must decode from independent updated application state data to a target updated data state, it can skip the inter-reference interdependent updated application state data between them to improve decoding speed. For the same example, if the decompressor can decode 50 compressed application states per second and there are two non-reference interdependent updated application states between every other context, the interval between independently updated application state data can be increased to 75, with the same 0.5-second response latency.

[0036] timing FIG. 8 is a timing diagram illustrating application state compression in a method for application state compression and replay according to an aspect of the present disclosure. The timing state diagram of FIG. 8 illustrates the interaction of elements of a device, including main memory (physical main memory), a processor, mass storage location 1, mass storage location 2, and mass storage location 3. Note that three mass storage locations are shown for illustrative purposes, and that a new mass storage location may exist for each instance of compressed application state data or compressed-updated application state data. Initially, as shown at 801, the processor may write application state data 802 to main memory. The processor may generate application state data by processing an application or by input entered by a user. Once application state data 802 is written to main memory, the processor may continue to process data, and in some implementations, a process switch may occur. Switching between processor applications may cause a processing pause, and the processor's memory management may swap the application state data from main memory. During the swap operation, as shown at 803, the processor may copy application state data 802 from main memory to a first mass storage location, as shown at 804. The first mass storage location may be a virtualized main memory storage space. Depending on the implementation after the swap operation, when the processor begins processing the application again, the application state data may be erased from main memory and copied back to main memory (not shown). Additionally, if the state memory is not completely swapped out due to the process switch, all pages may be forcefully swapped out at 803. After the application state data is copied to the first mass storage location, the device may begin compressing the application state data. The processor may compress the copied application state data 804, as shown at 805.Alternatively, another processing unit, such as, but not limited to, a GPU or a discrete compression processor or a discrete video CODEC, can perform the compression. In alternative embodiments, a GPU and / or CODEC accelerator may be used for decompression. Once compressed, the application state data is copied to a second location in mass storage at 806. In some embodiments, the second location in mass storage may be common storage instead of virtualized main memory.

[0037] After copying the compressed application state data, the device may resume processing of the application state data at 807. In some implementations, the processor may resume processing while the application state data is being compressed. In these implementations, the compression may be performed by a separate core or thread of the processor, or a GPU, or a discrete compression processor or a discrete video CODEC. Processing the application state data generates updated application state data 808, which is written to main memory. Again, the processor's memory management may initiate a swap operation in which the updated application state data 808 in main memory is copied at 809 to the virtualized main memory space in the first mass storage location, as shown at 810. Once copied to the mass storage location, the updated application state data may be compressed at 811 by the processor or another separate processing unit, such as a GPU, compression processor, or video CODEC. The compressed updated application state data 812 may then be written to a third mass storage location. Note that the previous compressed application state data was not overwritten. This allows previous application state data to be used for inter-predictive compression or as a starting point for different processing. In contrast, during swap operations, application state data or updated application state data held in the virtualized main memory space of the mass storage is erased and overwritten during each swap operation. Furthermore, each time compressed application state data or updated application state data is written to the mass storage, it may be written to a new location on the mass storage.

[0038] FIG. 9 is a timing diagram illustrating application state loading and replay in a method for application state compression and replay according to an aspect of the present disclosure. At some point during processing, the processor may reach a processing hardware pause, as shown at 902. This processing hardware pause may be determined by the application or selected by the user. For example, without limitation, an application may be programmed to use a compressed application state, such as a game save, when the application reaches an end state, such as a game over screen. Processing may be paused to allow the loading of a compressed application state or a compressed-updated application state. Alternatively, a user may be given the option to pause processing and resume the application from one of the compressed application state or the compressed-updated application state. As shown, while processing is paused, main memory may be populated with an in-process updated application state 901. A virtualized main memory space in a first mass storage location may be populated with updated application state data swapped in from main memory at 903. Elsewhere in the mass storage, compressed application state data 904 and updated compressed application state data 905 may be entered.

[0039] After processing is paused, the device may scan the mass store at 906 to determine the compressed application state 904 and the compressed updated application state 905 stored therein. The device may use the decompressed tags along with the compressed application state data and the compressed updated application state data to determine information about the compressed data 907. The information about the compressed application state 904 and the compressed updated application state 905 may be stored in main memory at 907. In some implementations, this information may include image frames or audio buffers reconstructed from the compressed application state 904 and the compressed updated application state 905. The device may display the information about the compressed application state and the updated application state on a display screen, as shown at 908. In some implementations, displaying the information about the compressed data may include displaying image frames reconstructed from the compressed data. Raw image frames may be stored along with the compressed data, and decompression may be applied to the compressed raw image frames to reconstruct image frames that can be displayed on a screen. Multiple image frames from the compressed application state data and the compressed updated application state data may be combined to reconstruct a video representation of the compressed stored state. The device may prompt the user to select an application state or an updated application state from the compressed data. The device may then wait 909 for the user to select an application state or an updated application state and begin replay.

[0040] When the processor receives a user selection between the compressed application state data and the compressed-updated application state data at 912, the processor can begin loading the selected state into main memory. To load the selected compressed application state data 910 or the compressed-updated application state data 911, the selected state is decompressed at 913. Before or after decompression, the suspended in-process state data in main memory may be (optionally) erased as shown at 914, 915. The selected decompressed application state data 910 or the selected decompressed-updated application state data 911 is written to main memory (physical main memory) at 916. Once the selected decompressed data 917 is written to main memory, the processor may begin processing 918 from the decompressed data in main memory. Alternatively, the decompressed data may be written to a new location in main memory, and the table pointer may be changed to point to the new location in main memory. The decompressed data in main memory represents a previous state during processing, and loading the decompressed data into main memory places the processor's data in that state during processing.

[0041] system 10 is a block diagram of a system implementing a method for application state compression and replay according to one aspect of the present disclosure. By way of example and not limitation, according to aspects of the present disclosure, system 1000 may be an embedded system, a mobile phone, a personal computer, a tablet computer, a portable gaming device, a workstation, a gaming console, etc.

[0042] System 1000 generally includes a central processing unit (CPU) 1002, a graphics processor unit (GPU) 1003, and memory 1004 accessible to both the CPU and the GPU. System 1000 may also include well-known support functions 1006 that may communicate with other components of the system, for example, via a data bus 1005. Such support functions may include, but are not limited to, input / output (I / O) elements 1007, a power supply (P / S) 1011, a clock (CLK) 1012, and a cache 1013. In addition to the cache 1013, GPU 1003 may include its own GPU cache, and the GPU may be configured to allow programs executing on GPU 1003 to read from or write to the GPU cache.

[0043] System 1000 may include a display device (not shown) for presenting rendered graphics to a user. In alternative implementations, the display device is a separate component that functions in conjunction with system 1000. The display device may be in the form of a flat panel display, a head-mounted display (HMD), a cathode ray tube (CRT) screen, a projector, or other device capable of displaying visible text, numbers, graphic symbols, or images.

[0044] The system 1000 also includes a mass storage device 1015, such as a disk drive, CD-ROM drive, flash memory, a solid-state drive (SSD), a tape drive, or the like, to provide non-volatile storage of programs and / or data. The system 1000 may also optionally include a user interface unit 1016 to facilitate interaction between the system 1000 and a user, for example, to provide input and control for applications. The user interface 1016 may include a keyboard, mouse, joystick, light pen, game controller, or other device that can be used in conjunction with a graphical user interface (GUI). The system 1000 may also include a network interface 1014 to enable the device to communicate with other devices over a network 1020. The network 1020 may be, for example, a local area network (LAN), a wide area network such as the Internet, a personal area network such as a Bluetooth network, or other type of network. These components may be implemented in hardware, software, or firmware, or some combination of two or more of these.

[0045] The processor 1002 and graphics processor 1003 may receive and use input from the user interface unit 1016, e.g., information corresponding to keystrokes, button presses, etc., while processing the application. The processor 1002 and graphics processor 1003 may optionally communicate with a remote client device 1021 through the network interface 1014 over the network 1020. The remote client device may send input, such as information corresponding to keystrokes, button presses, etc. The processor may use these inputs while executing the application. Image frames and other application data resulting from such execution may be transmitted to the client device, and application output executed by the processor 1002 and GPU 1003 may be streamed to the client device. The processor, acting as a remote host device in the methods shown in Figures 3, 4, and 9, receives a processing pause command from the remote client device 1021 and, in response, pauses processing and replays the state of the application. Additionally, the processor 1002 may transmit information regarding the compressed application state and updated compressed application state 1017 stored on the mass storage 1015, which may optionally include one or more reconstructed image frames and / or audio buffers from the compressed application state 1017. The processor may receive a selection from the remote client device 1021 of the compressed application state or the compressed updated application state 1017 in the mass storage 1015 to resume processing.

[0046] The CPU 1002 and GPU 1003 may each include one or more processor cores, e.g., a single core, two cores, four cores, eight cores, or more. In some implementations, the CPU 1002 and GPU 1003 may include multiple cores of the same accelerated processing unit (APU) 1001. If the CPU or GPU is a multi-core processor, underutilized cores may be used for state compression or decompression of stored state. For example, without limitation, free or underutilized cores may perform decompression while other cores process other uncompressed data. Underutilized cores may be used for compression while applications are running in parallel.

[0047] The memory 1004 may be in the form of an integrated circuit providing addressable memory, such as random access memory (RAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc. The memory 1004 may include dedicated graphics memory, which may store graphics resources and temporarily store graphics buffers of data for the graphics rendering pipeline. In the example shown in FIG. 10 , the graphics memory is shown as part of application state data 1008 in the main memory 1004. In alternative implementations, the graphics memory may be a separate hardware component, possibly integrated into the GPU 1003. The main memory 1004 may include application state data or updated application state data 1008 used by the processor 1002 and (optionally) the GPU 1003 during processing. Application state data may be a snapshot of the state of data being processed by the processor that has been moved or copied from registers to main memory, and updated application state data may be modified application state data that has not yet been moved from main memory to mass storage, also referred to as dirty pages. Register data is the most volatile data. As a first step to obtain updated application state data, register data is copied to memory 1004. As mentioned above, when the "dirty page" flag is used, it is desirable to synchronize memory swap operations with application state compaction operations. Application state data and updated application state data 1008 may be stored in memory pages of memory 1004.During processing, the processor may pause processing and swap pages of application state data 1008 from main memory 1004 into a virtualized main memory space located in mass storage 1015, so that in some embodiments uncompressed application state data or uncompressed updated application state data 1018 may be stored in a virtualized main memory space in mass storage 1015. This virtualized main memory space appears to the processor to be part of main memory despite being located in mass storage random access memory 1015.

[0048] According to aspects of the present disclosure, during processing, as discussed in FIGS. 1, 2, and 8, the application state or updated application state 1008 from main memory may be compressed, and the compressed application state or compressed updated application state 1017 may be stored in a location in mass storage 1015. Additionally, main memory may include a compression program or algorithm 1009 for use by the processor. In some alternative implementations, the graphics processor 1003 may perform the compression. Or, in some other alternative implementations, another discrete processor, such as, but not limited to, a discrete video CODEC 1023, may perform the compression. Mass storage 1015 may include application or program data 1022 that is loaded into main memory 1004 when processing begins on the application or program 1022. As shown in FIG. 5, memory pages in main memory may include application context data and may include application state data created by the processor during processing of the application data 1022. Main memory 1004 may also include compressed application state data 1017 stored in mass storage 1015 as shown in Figures 3, 4, and 9, and application state selection data 1010, which may include information regarding application replay selection and loading. In some implementations, application state selection 1010 data may also include one or more frames reconstructed from the compressed application state data or the compressed updated application state data 1017.

[0049] By way of example and not limitation, the CPU 1002 and GPU 1003 may access memory 1004 via one or more buses 1005. In some cases, it may be useful for the system 1000 to include two or more different buses. The memory 1004 may contain data that the CPU 1002 and GPU 1003 can access. The GPU 1003 may include multiple compute units configured to perform graphics processing tasks in parallel. Each compute unit may include its own dedicated local memory store, such as a local data share. Alternatively, the compute units may each access the memory 1004 or a dedicated graphics memory.

[0050] As used herein and generally understood by those skilled in the art, an application specific integrated circuit (ASIC) is an integrated circuit that is customized for a particular application, rather than for general use.

[0051] As used herein and generally understood by those skilled in the art, a field programmable gate array (FPGA) is an integrated circuit that is designed to be configured by a customer or designer after manufacture - hence "field programmable." FPGA configurations are generally specified using a hardware description language (HDL) similar to those used for ASICs.

[0052] As used herein and as commonly understood by those skilled in the art, a system on a chip or system-on-chip (SoC or SOC) is an integrated circuit (IC) that integrates all the components of a computer or other electronic system onto a single chip. This can include digital, analog, mixed-signal, and often radio frequency functionality—all on a single chip substrate. A typical application is in the field of embedded systems.

[0053] A typical SoC includes the following hardware components: One or more processor cores (e.g., a microcontroller, microprocessor, or digital signal processor (DSP) core). Memory blocks, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM) and flash memory. A timing source, for example, an oscillator or a phase-locked loop. Peripherals, such as counter timers, real-time timers, or power-on reset generators. External interfaces, such as industry standards such as Universal Serial Bus (USB), FireWire®, Ethernet, universal asynchronous receiver / transmitter (USART), and Serial Peripheral Interface (SPI) bus. Analog interfaces (including analog-to-digital converters (ADCs) and digital-to-analog converters (DACs)). Voltage regulators and power management circuits.

[0054] These components are connected by either proprietary or industry-standard buses. Direct Memory Access (DMA) controllers increase the data throughput of the SoC by routing data directly between external interfaces and memory, bypassing the processor core.

[0055] A typical SoC includes both the hardware components described above and executable instructions (eg, software or firmware) that control the processor core(s), peripherals, and interfaces.

[0056] Aspects of the present disclosure allow for quick access to a previous execution state of an application and restarting the application from a selected previous state. Such functionality has application to video games and other forms of computer simulations, such as flight simulators, driving simulators, and ship navigation simulators.

[0057] While the above is a complete description of the preferred embodiment of the present invention, it is possible to use various alternatives, modifications, and equivalents. Therefore, the scope of the invention should be determined not with reference to the above description, but instead with reference to the appended claims, along with their full scope of equivalents. Any feature described in this specification, whether preferred or not, may be combined with any other feature described in this specification, whether preferred or not. In the following claims, the indefinite article "A" or "An" refers to a quantity of one or more of the item following the article, unless expressly stated otherwise. The appended claims should not be construed as including means-plus-function limitations unless such limitations are expressly recited in a given claim using the phrase "means for."

Claims

1. A computer-implemented method for replaying an application, comprising: compressing application state data from main memory; writing the compressed application state data to a first location in mass storage; generating updated application state data; compressing the updated application state data from the main memory; writing the updated application state data to a second location of the mass storage; pausing processing of the application state data and the updated application state data; scanning the compressed application state data and the compressed updated application state data stored in the mass storage; using information from the scanned compressed application state data and the compressed updated application state data to display information corresponding to the compressed application state data and the updated compressed application state data stored in the mass storage; and reconstructing an image frame or an audio buffer from the compressed application state data or the compressed updated application state data.

2. A computer-implemented method for replaying an application, comprising: compressing application state data from main memory; writing the compressed application state data to a first location in mass storage; generating updated application state data; compressing the updated application state data from the main memory; writing the updated application state data to a second location of the mass storage; pausing processing of the application state data and the updated application state data; scanning the compressed application state data and compressed updated application state data stored in the mass storage; and using information from the scanned compressed application state data and the compressed updated application state data to display information corresponding to the compressed application state data and the updated compressed application state data stored in the mass storage; generating the updated application state data includes processing two or more image frames, and compressing the updated application state data includes compressing the updated application state data after processing the two or more image frames.

3. Decompressing the selected compressed application state data or the selected compressed updated application state data stored in the mass storage; writing the deployed application state data or the deployed updated application state data to the main memory; and The method of claim 1 , further comprising processing the deployed application state data or the deployed updated application state data in main memory.

4. 2. The method of claim 1, wherein displaying information corresponding to the compressed application data and the updated application data includes displaying an image frame reconstructed from the compressed application state data or the compressed updated application state data.

5. 2. The method of claim 1, wherein displaying information corresponding to the compressed application data and the updated application data includes playing a buffer of audio reconstructed from the compressed application state data or the compressed updated application state data.

6. 2. The method of claim 1, wherein compressing the application state data or compressing the updated application state data further comprises determining a compression type to apply to the application state data or the updated application state data.

7. The method of claim 6 , wherein determining the compression type includes determining whether to apply run-length coding, dictionary compression, or a general compression algorithm.

8. 8. The method of claim 7, wherein the general compression algorithm comprises Context Tree Weighting (CTW), and the dictionary-based compression comprises Lempfle-Ziv-Welsh compression, Lempfle-Ziv-Huffman-Arithmetic-Markov (LZHAM) compression, or DEFLATE compression.

9. The method of claim 6 , wherein compressing the updated application state data includes determining whether to apply cross-prediction to the updated application state data.

10. The method of claim 9 , wherein applying cross-prediction to the updated application state data comprises performing an exclusive-or between the application state data and the updated application state data.

11. The method of claim 10 , wherein applying inter-prediction to the updated application state data further comprises lexicographically compressing the inter-prediction and sharing a compression dictionary with the compressed application state data.

12. The method of claim 2 , wherein compressing the updated application state data comprises compressing the updated application state data after a set interval of image frames has been processed.

13. The method of claim 12 , wherein the set interval image frames are defined by a user or by the application state data.

14. The method of claim 1 , wherein the application state data and the updated application state data include image frames and application context.

15. The method of claim 14 , wherein the application context includes processor registers, a stack, and a heap.

16. The method of claim 14 , wherein the application state data further comprises emulated device state data.

17. 2. The method of claim 1, wherein displaying information corresponding to the compressed application state data and updated compressed application state data stored in the mass storage further comprises transmitting the information to a remote client.

18. 20. The method of claim 17, wherein the processing is paused in response to a request from a remote client.

19. 4. The method of claim 3, further comprising clearing suspended in-process or in-process updated application state data from the main memory before writing the deployed or deployed updated application state data to the main memory.

20. 4. The method of claim 3, wherein processing the unpacked or updated application state data in main memory includes resuming the application at a point in the application's processing where the application state data or the updated application state data was generated before compression.

21. 2. The method of claim 1, comprising copying the application state data from physical main memory to a virtual main memory space before compressing the application state data, and further comprising copying the updated application state data from the physical main memory to the virtual main memory space before compressing the updated application state data.

22. 22. The method of claim 21, wherein the virtual main memory space includes a portion of non-volatile memory reserved for the virtual main memory space.

23. 1. A system for application replay, comprising: processor, a main memory connected to said processor; a mass storage device connected to the main memory and the processor; When executed by the processor, compressing application state data from said main memory; writing the compressed application state data to a first location in the mass storage; generating updated application state data; compressing the updated application state data from the main memory; writing updated application state data to a second location of said mass storage; pausing processing of the application state data and the updated application state data; scanning the compressed application state data and the compressed updated application state data stored in the mass storage; using information from the scanned compressed application state data and the compressed updated application state data to display information corresponding to the compressed application state data and the updated compressed application state data stored in the mass storage; and The system includes instructions embodied in the main memory that cause the processor to execute a method for replaying an application, the method including reconstructing an image frame or an audio buffer from the compressed application state data or the compressed updated application state data.

24. A system for replaying an application, comprising: processor, a main memory connected to said processor; a mass storage device connected to the main memory and the processor; When executed by the processor, compressing application state data from said main memory; writing the compressed application state data to a first location in the mass storage; generating updated application state data; compressing the updated application state data from the main memory; writing updated application state data to a second location of said mass storage; pausing processing of the application state data and the updated application state data; scanning the compressed application state data and compressed updated application state data stored in the mass storage; and using information from the scanned compressed application state data and the compressed updated application state data to display information corresponding to the compressed application state data and the updated compressed application state data stored in the mass storage; 11. A system comprising: instructions embodied in the main memory that cause the processor to execute a method for replaying an application, wherein generating the updated application state data comprises processing two or more image frames; and compressing the updated application state data comprises compressing the updated application state data after processing the two or more image frames.

25. A non-transitory computer-readable medium having executable instructions embodied therein that, when executed by a computer, compressing application state data from main memory; writing the compressed application state data to a first location in mass storage; generating updated application state data; compressing the updated application state data from the main memory; writing updated application state data to a second location of said mass storage; pausing processing of the application state data and the updated application state data; scanning the compressed application state data and the compressed updated application state data stored in the mass storage; using information from the scanned compressed application state data and the compressed updated application state data to display information corresponding to the compressed application state data and the updated compressed application state data stored in the mass storage; and A non-transitory computer-readable medium that causes the computer to perform a method for replaying an application, the method including reconstructing an image frame or an audio buffer from the compressed application state data or the compressed updated application state data.

26. A non-transitory computer-readable medium having executable instructions embodied thereon, which when executed by a computer: compressing application state data from main memory; writing the compressed application state data to a first location in mass storage; generating updated application state data; compressing the updated application state data from the main memory; writing updated application state data to a second location of said mass storage; pausing processing of the application state data and the updated application state data; scanning the compressed application state data and compressed updated application state data stored in the mass storage; and using information from the scanned compressed application state data and the compressed updated application state data to display information corresponding to the compressed application state data and the updated compressed application state data stored in the mass storage; 12. A non-transitory computer-readable medium that causes a computer to perform a method for replaying an application, wherein generating the updated application state data includes processing two or more image frames, and compressing the updated application state data includes compressing the updated application state data after processing the two or more image frames.

Images