Application process context compression and playback

Abstract

Application state data from the main memory may be compressed, and the compressed data may be written to a first location in the mass storage. Updated application state data is generated, and the updated application state data from the main memory is compressed. The updated application state data is then written to a second location in the mass storage. Processing of the application state data and the updated application state data may then be suspended. The compressed application state data and the compressed and updated application state data stored in the mass storage are scanned; and displaying information corresponding to the compressed application state data and the updated compressed application state data stored in the mass storage using information from the scanned compressed application state data and the compressed updated application state data.

Description

Technical Field

[0001] Aspects of the present disclosure relate to memory management, and in particular to application replay from compressed main memory pages. Background of the Invention

[0003] Modern computer architectures use main memory to store data on which the processor operates. The processor can transfer information from main memory to its registers to perform operations, and then transfer the register contents back to main memory. Main memory also contains application state data, which is a snapshot of the state of the data being processed by the processor moved from registers to main memory. Application state data can include application context, such as processor registers, stacks, and heaps. Main memory data is periodically saved to non-volatile memory, also known as mass storage, such as a hard disk drive (HDD) or solid-state drive (SSD). During processing, the processor can update the application state data in main memory without updating the data stored in mass storage; this is sometimes called a dirty page because the page data has been updated but not written to mass storage. Application state data is stored in one or more pages of main memory, and each page consists of multiple fixed-size page frames or variable-size page segments. The processor maintains an address buffer that holds the location of the application state data within main memory.

[0004] Many operating systems implement memory swapping. This allows the operating system to increase the size of main memory without having additional physical memory. Memory swapping uses a portion of non-volatile memory storage space as main memory. While application processing is paused, swapping pulls application state data from main memory and writes it to space in non-volatile memory. When changing the application being processed, the operating system can swap the application state data written to that area of ​​non-volatile memory back and forth between physical memory in main memory and that area of ​​non-volatile memory. Swapping pages from physical memory to non-volatile storage and restoring pages from non-volatile memory to physical memory are controlled by two different algorithms. When an application tries to allocate a page of memory, but physical memory is full, the operating system will suspend the application and pick a page that is not actively being used and copy it to non-volatile memory. The operating system will then reuse the page for memory allocation and resume the application. Another reason for swapping pages out is application context switching. If an application is switched out, its memory pages are also swapped out to make room for the application being swapped in.

[0005] It has been proposed that by compressing application data stored in portions of main memory, the efficiency of memory management can be improved. Various compression techniques have been proposed to reduce the size of application state data stored in main memory. These techniques have not yet been adopted by the industry as a whole.

[0006] Video games and other applications use separate points programmed within the application to save state data for the application. These are often referred to as 'game saves', 'save states' or 'save files'. These save files may be updated or created periodically by the game application, or at the will of the user depending on the implementation. The creation of save data operates independently of processor memory management, and save data is created on mass storage independently of main memory or memory swapping. The content of the save data varies depending on the implementation and is not standardized between applications.

[0007] It is within this context that aspects of the present disclosure are presented. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The teachings of the present disclosure may be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:

[0009] Figure 1 is a timing diagram illustrating application state compression according to an aspect of the present disclosure.

[0010] Figure 2 is a flow chart illustrating application state compression in a method for application state compression and replay according to an aspect of the present disclosure.

[0011] Figure 3 is a flow chart depicting application state selection in a method for application state compression and replay according to an aspect of the present disclosure.

[0012] Figure 4 is a flow chart illustrating application state replay loading in a method for application state compression and replay according to an aspect of the present disclosure.

[0013] Figure 5 is a block diagram illustrating the contents of a memory page containing an application state according to an aspect of the present disclosure.

[0014] Figure 6 is a flow chart illustrating application state compression in a method for application state compression and replay according to an aspect of the present disclosure.

[0015] Figure 7 is a flow chart depicting updated application state compression in a method for application state compression and replay according to an aspect of the present disclosure.

[0016] Figure 8 is a timing diagram illustrating application state compression in a method for application state compression and replay according to an aspect of the present disclosure.

[0017] Figure 9 is a timing diagram depicting application state loading and replay in a method for application state compression and replay according to an aspect of the present disclosure.

[0018] Figure 10 is a schematic block diagram of a system implementing a method for application state compression and replay according to an aspect of the present disclosure. DETAILED DESCRIPTION

[0019] Although the following detailed description contains many specific details for illustrative purposes, it will be appreciated by any person skilled in the art that many variations and modifications to the following details are within the scope of the present invention. Therefore, the exemplary embodiments of the present invention described below are set forth without losing the generality of the claimed invention and without imposing limitations on the claimed invention.

[0020] Although it has been shown that page compression can be used to improve memory management efficiency, these techniques have not been widely adopted due to a lack of hardware resources. Now, fast multi-core processors are common, and hardware resources are sufficient to implement page compression without affecting processing. Memory management systems and other applications have not yet taken advantage of memory compression.

[0021] Although previously proposed memory compression methods involve compressing memory pages directly to main memory during processing, according to various aspects of the present disclosure, more powerful system functions can be obtained by compressing memory pages and retaining compressed memory pages in mass storage. Retaining compressed memory pages in mass storage allows application replay, and through replay, the user can roll back the processing of the application to a previous point during processing. For example, but not limited to, retaining compressed memory pages in mass storage can allow a video game to be replayed from a previous point within the video game without saving the game, or restarting the simulator program to a certain point during the simulation without saving the state, or undoing changes made in a drawing or writing program that did not save the previous state, or the retained compressed application state can be checked in program debugging to determine the application context and find vulnerabilities, or the retained compressed application state can be used to recover lost application data after a system crash. The benefits of retaining compressed application state are many, and allow functions to be added to programs that previously lacked the ability to save state or had infrequent save points. In addition, the compressed application state includes more information than the save file created by the application, and can allow finer-grained application replay or more user-friendly application replay points.

[0022] Method

[0023] Figure 1An implementation of application state compression according to one aspect of the present disclosure is depicted. In this particular implementation, an application (e.g., a game application) performs processing at predetermined time intervals and generates output in the form of data corresponding to an image frame at the end of each interval. However, aspects of the present disclosure are not limited to only such an implementation. As shown, initially, the system may perform processing and generate a first image frame 101. After the first image frame is generated, the system pauses at 102 and copies the paged application state data in the main memory to the virtualized main memory space at a first location in the mass storage at 103. The application state data in the main memory page is then compressed at 104 and copied to a second location in the mass storage at 105. As used herein, and as generally understood by those skilled in the art, "state" is a general term that means all necessary information for defining a system. Application "context" more specifically refers to all necessary information for resuming a computer application.

[0024] The processor resumes processing the application state data and generates a second image frame 106. The generation of the second image frame 106 may include generating updated application state data stored in the main memory. In some implementations, the application process may be paused at 107, and the updated application state data generated using the second frame may be copied from the physical main memory to the virtual main memory at the first location in the mass storage at 108. The application state data generated using the second frame may then be compressed at 109 and copied to a third location in the mass storage at 110.

[0025] In an alternative implementation, one or more image frames may be processed using the corresponding updated application state data or application data without compressing the memory pages of the updated application state data. Therefore, elements 107 to 110 are shown in dashed lines, indicating that the operation is optional depending on the implementation. In another alternative implementation, the application state data copied at 103 and the updated application state data copied at 108 may also be stored in physical memory. The copied pages corresponding to the state data are snapshots of the application. Memory compression can be accomplished with snapshots in physical memory.

[0026] As shown, a third image frame 111 may be generated using the updated application state data. Processing may then be paused at 112 and a memory page corresponding to the updated application state data may be copied to a virtual main memory space at a first location in mass storage at 113. The updated application state data generated using the third image frame may then be compressed at 114 and copied to a fourth location in mass storage. Alternatively, if the previous updated application state data was not compressed, the compressed updated application state data may be copied to a third location in mass storage. This process may continue until an nth image 115, where n is the number of image frames generated while running the application.

[0027] like Figure 1 As shown, in some implementations, the pause in processing may last only long enough to copy the application state data from the main memory to the virtual main memory space. According to some aspects of the present disclosure, parallel processing with a processor (e.g., a second thread, core, or processing unit), a GPU, a discrete codec, etc. may be used to compress and copy the application state data to a mass storage device in addition to the swap operation. In some implementations, a processor other than a CPU (e.g., a GPU) may be used to decompress the compressed application state data or the compressed updated application state data.

[0028] In addition, it should be noted that in some cases, the processed application state data can be saved to mass storage independently of the compressed paging method currently disclosed. For example, as described above, the operating system can swap out some inactive pages from physical memory. If these pages are "clean", that is, the state has not changed since the previous interval or frame, they can be compressed and stored directly without restoring the page back to physical memory. However, if a page is "dirty", that is, the state has changed since the previous interval or frame, the latest memory content from physical memory must be selected and stored. In this case, the "clean page" flag and the "dirty page" flag can be used as indicators to help the memory compressor. If a page can remain clean after a processing interval (e.g., a frame iteration), the compressor can remove the page from the current frame context and only use an index to refer to the page to the previous frame context. Otherwise, the dirty page is stored with the context of the current interval (e.g., the current frame). When the "dirty page" flag is used, it is expected that the page memory swap should be synchronized with the application state compressor operation.

[0029] If the application state compressor runs in conjunction with virtual memory management, memory page swaps can be performed synchronously with application state compression. In other words, the operating system will only perform page memory swaps when the application compressor requires that page memory to be compressed. For this case, the dirty page flag means that the page has changed between the application state data and the updated application state data. Alternatively, if the operating system swaps memory pages independently of the application state compressor, then dirty pages are only pages that have changed since the last page swap, but the last swap was timed independently of the application state data compression and the updated application state compression. In this case, the dirty page flag is not useful for determining whether the data on the flagged page has changed, and the application compressor should always assume that the data has changed.

[0030] As mentioned above, aspects of the present disclosure are not limited to Figure 1 The implementation described in . Figure 2 Details of application state compression according to one aspect of the present disclosure are more generally shown. The method for application state compression and replay may begin with selecting an application 201. The processor may pull application data from mass storage to main memory for processing. The processor may then process the application and generate application state data. After pausing processing, the application state data is copied from a register of the processor to a page of main memory, as indicated by 202. In some optional implementations, the processor may perform a swap operation at 203 by copying the application state data from physical main memory to a first location (e.g., a virtual memory space in mass storage or a location in main memory). The processor then reads and compresses the application state data from the first location, as indicated by 204. In some alternative implementations, the swap operation may be skipped, and instead the processor may read and compress the application state data directly from main memory at 204, so compression of the application state does not require a swap operation. Next, the compressed page of the application state data is written to a second location in mass storage, as indicated by 205. The second location in the mass storage is outside of the virtualized main memory space, and thus the processor does not operate in this area of ​​the mass storage as it does in main memory. After processing is suspended, the processor may resume processing the application state data at 206 and process one or more image frames from the application state data, creating updated application state data that is stored as pages in the physical main memory.

[0031] As before, the processor is paused and the updated application state data may be (optionally) swapped from a page in physical main memory to virtual main memory space at a first location in mass storage, at 207. The updated application data is then compressed at 208 and written to a third location in mass storage at 209. It should be noted that compression of the updated application state data need not be performed at every processing interval, e.g., after every image frame, and may occur at intervals such as every 2 frames, every 3 frames, every 5 frames, or every n frames, where n is determined by the application, the operating system, or based on available resources. Alternatively, n may be determined by the user.

[0032] The processor may continue to process the application state data and generate image frames and updated application state data at 210, or the application may reach an exit processing and compressed application state, as indicated at 211. The exit state may occur when processing reaches an end state or a program interruption, alternatively, the exit state may be a user-selected pause in application processing to return to a previous application state or an updated application state.

[0033] For many use cases, it is desirable to synchronize page swapping out with application state compression. Instead of swapping out pages when physical memory is full or when an application switches, pages can be periodically copied from one location to another for application state compression.

[0034] Furthermore, since swapped memory pages are compressed before being stored, standard operating system procedures can be modified to store swapped pages in another location in physical memory rather than in non-volatile memory. It is often easier for data compressors to work on physical memory. Only compressed data is stored in non-volatile memory.

[0035] As described above, memory page swapping may be performed synchronously with application state compression, or may be performed independently of application state compression.As described above, clean and dirty page flags may be used in conjunction with such implementations.

[0036] Figure 3is a block diagram depicting application state selection in a method for application state compression and replay according to one aspect of the present disclosure. Initially, the processor may enter the application state selection method from the application state compression method due to an interruption in application processing (such as a program interruption or application failure) or due to user input. The processor suspends processing of the current application state data, as indicated at 301. Then, the system may scan a large capacity storage device at 302 to find compressed application state data and updated application state data. In some optional implementations, the system may reconstruct image frames associated with one or more states from the application state data or the updated application state data at 303. In addition, in some other alternative implementations, audio associated with one or more states from the application state data may be reconstructed from an audio buffer in the application state data or the updated application state data at 303. The audio may be played through, for example, but not limited to, a speaker, and may be played with or without displaying an image frame. According to aspects of the present disclosure, some implementations of the application state data and the updated application state data may include associated image frames and / or audio buffers compressed together with the application state data or the updated application state data. Decompressing the compressed image frames and / or audio buffers in the application state data or updated application state data can reconstruct the image frames for display or reconstruct the audio for playback. Alternatively, the application context can be processed to reconstruct the image frames or audio buffers. Using information from the scanned compressed application state data and the compressed updated application state data, the system can display or playback information corresponding to the compressed application state data and the compressed updated application state data in the mass storage at 304. In this optional implementation, the information can include reconstructed image frames or audio buffers generated from the compressed application state data or the compressed updated application state data.

[0037] In some implementations, a list of compressed application state data and compressed updated application state data may be displayed, which may be sorted in chronological order. To facilitate quick chronological organization of the display, each instance of the compressed application state data and the compressed updated application state data may include a label that describes such attributes as a creation date and time and an application instance. A display of information corresponding to the compressed application data and the compressed updated application data may be generated by, for example, but not limited to, displaying the information in the label on a display screen. In some implementations, this may also include displaying a reconstructed image frame and / or playing back an audio buffer reconstructed from the compressed data. Once the information corresponding to the compressed application state data and the compressed updated application state data is displayed, the processor waits for a selection made by the user at 305. In some alternative implementations, image frame data from the compressed application state data or the compressed updated application state data may be displayed at 306 according to the user's selection. In some implementations, audio buffer data from the compressed application state data or the compressed updated application state data may be played back at 306 according to the user's selection with or without displaying the image frame. This implementation can reduce wasted decompression cycles while still providing functionality to the user, providing an intuitive idea of ​​the selected state of the application. Before the user confirms that he wants to restart the application from the selected compressed application state or compressed updated application state, the decompressed image frames and / or audio buffers can be rebuilt. Once the user selects or confirms the selected compressed application state or compressed updated application state, the system can start the application using the compressed application state data or compressed updated application state data at 307, for example, as described below with respect to Figure 4 described.

[0038] Figure 4The flowchart depicted in shows application state replay loading in a method for application state compression and replay according to an aspect of the present disclosure. After selecting a compressed application state or a compressed updated application state at 401, the system will initialize processing from the compressed application state or the compressed updated application state. First, the compressed application state or the compressed updated application state is decompressed, as indicated by 402. In some implementations, the compressed application state data or the compressed updated application state data is copied to a memory, and the processor performs operations in the memory to decompress the data. In alternative implementations, a separate processing unit (such as a graphics processor) or a discrete compression / decompression circuit (such as a video encoder / decoder (CODEC)) can perform decompression, and the compressed application state data or the compressed updated application state data can be copied to a memory space reserved for the separate processing unit. To decompress the compressed application state data or the updated application state data, the system can use the inverse of the algorithm used to compress the data to decompress the data, which can include using the same dictionary used for dictionary encoding, using the inverse operation of a general compression algorithm, or applying X-OR data to a base data set for inter-frame prediction data. In some optional implementations, the application state data in the main memory for which the process is suspended may be erased at 403 before, during, or after decompression to make room for the decompressed application state data or the decompressed updated application state data. Alternatively, a portion of the application state data in the main memory for which the process is suspended may be retained, and when the decompressed application state data or the decompressed updated application state data is copied to the main memory, one or more pointers to the corresponding memory location may be changed to the location of the data. Once the compressed application state data or the compressed updated application state data is decompressed, it may be copied to the main memory at 404. As discussed, the decompressed-compressed application state data or the decompressed-compressed updated application state data may be copied to the same location as the application state data or the updated application state data in the previous process. Alternatively, as described above, the decompressed-compressed application state data or the decompressed-compressed updated application state data may be copied to a new location in the main memory, and pointers in different tables (such as a translation lookaside buffer) may be changed. The processing of the application can be resumed in the same manner as a normal context switch, such as from one application to another. Specifically, the conventional context switch process may include a process of loading registers from memory to the processor before starting the processor. Such a loading process (sometimes referred to as initialization) may be performed before the processing is resumed.Finally, after copying the decompressed-compressed application state data or the decompressed-compressed updated application state data to main memory and performing any necessary initialization, the processor may unpause and begin processing the copied state data in main memory at 405 .

[0039] Thus, in practice, according to certain aspects of the present disclosure, a game player may use a captured video of the game play (reconstructed from the compressed application state data and the compressed updated application state data) to search for replay points. The game video may be played forward or reverse. When the game player finds a good replay point, he or she may pause the video and start replaying from that point. To enable this use case, a series of time codes may be embedded as tags into the compressed application state data and the compressed updated application state data. If the game player pauses the video at a particular frame, the video frame time code may be passed to the decompression hardware and / or software to retrieve the application state or updated application state for that time code. NOTE: The game video frame rate may be different than the application context capture rate. The time codes between the captured context and the captured video may not match one to one.

[0040] Compression

[0041] Figure 5The block diagram shown in depicts an example of the contents of a memory page containing application state according to an aspect of the present disclosure. As shown, a memory page 501 is composed of a plurality of fixed-size frames 502. In some alternative implementations, a memory page may be composed of a plurality of variable-size segments, each segment size of which may be recorded in a table. Application state data and updated application state data 507 may span multiple frames or segments in a memory page 501. In some implementations, the application state data and updated application state data may include both image frame data and / or audio buffer data 503 and associated application context data 504. Alternatively, the application state data or updated application state data may be application context data without an associated image frame or audio buffer. It should be noted that an image frame is data representing pixel values ​​for display on a display screen. An audio buffer is data representing audio signal samples for playback on a speaker or other sound generating device. Application context data may include application data, add host processor and GPU registers, GPU memory and GPU compute unit local memory, stacks, and heaps. A stack is a linear data structure that tracks variables used by a processor during processing in an ordered manner. The heap is a large unordered memory space for other variables and memory objects created during processing. The memory page may also contain other data such as data 506 of other applications or operating system processes. Although the memory page 501 is shown as being divided into multiple frames, the memory page may be divided into smaller memory blocks. Larger sizes of compressed memory blocks may improve compression efficiency, but smaller sizes of blocks may be compressed faster. The compressor may determine a suitable compression model for each block. Statistically, the larger the block size, the better the compression ratio. In some implementations, the compressed memory blocks may be of variable size and have tags or tables to present the characteristics of the memory blocks, such as but not limited to size and location. In some implementations, after compression, the compressed application state data and the compressed updated application state data may include uncompressed tags that describe the attributes of the data, such as creation time, application, and size. Tags may also indicate which portion of the compressed data is an image frame or an audio buffer. Tags may be compressed or uncompressed. Uncompressed tags have faster retrieval speeds. Compressed tags have better compression efficiency.

[0042] Figure 6The flowchart shown in shows an example of application state compression in a method for application state compression and replay according to an aspect of the present disclosure. Initially, the application state data can be modeled to find exploitable regularities within the data in a given page, block, segment, or other data unit, as indicated at 601. Modeling can determine regularities such as the number of zeros in the data, the repetition of certain data strings or structures, and the like. After modeling, the device can determine the best compression algorithm by applying different compression algorithms 605, 606, 603 and measuring the compression 602 of the resulting compressed application state data. The compression can be measured using the final memory size of the compressed application state data. The memory sizes of the compressed application state data created by the general compression 605, dictionary compression 606, and run-length compression 603 can be compared at 604 to determine the compressed application state data with the smallest memory size. The compressed application state data of the smallest size can then be stored in a mass storage 607. In some alternative implementations, the final compressed file size can be used to obtain the compression ratio. The compression ratio can be the ratio of the memory size of the compressed application state data to the memory size of the uncompressed application state data. Compression ratios may be generated for a generic compression algorithm 605, a word-typical compression algorithm 606, and run-length encoding 603. The ratios of all applied compression algorithms may be compared at 604, and the algorithm resulting in the lowest memory size may be selected and stored in mass storage 607.

[0043] If it is determined from modeling and initial compression that the application state data unit in question contains many zero digits or single digit repetitions, then run-length encoding can be applied to that data unit stored in mass storage. Run-length encoding takes each single digit repetition and reduces it from many repetitions of a single digit to a two-entry encoding that provides the digit and the number of repetitions in sequence. For example, but not limited to, a sequence such as AAAAABBBBCCDDDDD encoded in a run-length encoding manner would be A5B4C2D5, thereby reducing a sequence of 16 characters to 8.

[0044] Modeling data and initial compression can determine whether there are any exploitable regularities within the data based on the size or compression ratio of the compressed application state data. If the application state data includes exploitable regularities, dictionary encoding such as Lemple-Ziv-Welch series (LZW) compression, DEFLATE compression, or Lemple-Ziv-Huffman-Arithmetic-Markov (LZHAM) compression can be applied to the compressed application state data stored in the mass storage. Dictionary encoding can compress the application state data by replacing regularities within the application state data with tokens and placing the tokenized regularities in a table (dictionary). If the regularities within the data are insufficient to be encoded using a dictionary compression method (as determined by modeling 601 and comparing the compressed memory size 604 from the initial application of the compression algorithm), the final compressed application state data stored in the mass storage 607 can be compressed using another general compression algorithm, such as, for example, but not limited to, context tree weighted (CTW) encoding or other similar lossless compression algorithms.

[0045] Figure 7 701 is a block diagram depicting updated application state compression in a method for application state compression and playback according to an aspect of the present disclosure. Initially, memory pages of updated application state data are modeled to determine regularities and similarities between the updated application state data and previously generated application state data, as indicated at 701. After modeling, the updated application state data can be compressed by applying compression algorithms such as inter-frame prediction and general compression 708, inter-frame prediction and dictionary compression 707, inter-frame prediction and run-length coding 705, and Figure 6703) to determine the best compression algorithm. Compression 702 can be measured using the final memory size of the compressed application state data generated by each algorithm 708, 707, 705, and 703. The memory size of the compressed application state data created by each algorithm 708, 707, 705, and 703 can be compared at 704 to determine the compressed application state data with the smallest memory size. The compressed application state data with the smallest memory size can then be stored in a mass storage 709. In some alternative implementations, the final compressed file size can be used to obtain a compression ratio. The compression ratio can be the ratio of the memory size of the compressed application state data to the memory size of the uncompressed application state data. A compression ratio can be generated for each of the compression types 708, 707, 705, and 703. The ratios of the compression algorithms of all applications can be compared at 704, and the algorithm that results in the lowest memory size can be selected and stored in the mass storage 709. The device may use data modeling and the compressed state data memory size to determine whether there are any similarities between the updated application state data and the previous application state data. If there are no similarities, the device may compress the updated application state data using a de-novo compression algorithm (i.e., an algorithm without inter-frame prediction), such as described above and as Figure 6 As shown. If there is a similarity between the updated application state data and the previous application state data determined by modeling 701 and applying inter-frame prediction and compression, the device uses data modeling and the compressed data size and determines that there are a large number of zero digits or single digit repetitions in the data. When many single digits or zero repetitions are found, inter-frame prediction and run-length coding can be applied to the updated application state data. Inter-frame prediction performs an exclusive OR (XOR) on the updated application state data and the previous application state data, thereby creating a data page that only contains the difference between the updated application state data and the previous application state data. Run-length coding can be applied to the changed data page created by the XOR operation and stored in mass storage 709 based on the comparison of the compressed memory size at 704. As described above, run-length coding then removes the single digit repetition string using two entry symbols. If the number of single digit repetitions is insufficient, the device can determine that there is a pattern in the data by applying a compression algorithm and the comparison at 704, and dictionary compression can be used for the stored compressed state data. If a sufficient number of regularities exist within the updated application state data as determined by applying the compression algorithm and comparing 704, inter-frame prediction and dictionary coding may be used to compress the stored updated application state data, such as using LZW. When paired with inter-frame prediction, dictionary coding may use the same coding table (dictionary) used for previous application state data.

[0046] If there is not a sufficient number of regularities within the updated application state data, inter-frame prediction and a common encoding method may be applied to the stored compressed state data. As described above, the common encoding method may be a lossless compression method such as, but not limited to, CTW. In some implementations, there may be many updated application state data, but only a small number of 'clean pages' of application state data. Thus, to allow random access, independent updated application state data may be inserted and compressed as application state data (e.g., at intervals based on the speed of decompression) at a time. Figure 6 ). For example, but not limited to, if 50 compressed updated application states can be decompressed per second, then compressed independent updated application state data can be created after every 25 compressed updated application states. The implementation in this example will result in a maximum response time of half a second. If the selected updated application state data is close to the independent updated application state data, the response time may be faster. The independent updated application state data includes all memory pages, not just dirty memory pages. In addition, if according to Figure 6 The method shown encodes the independent updated application state data, and does not apply inter-frame prediction to the independent updated application state data. Inserting the independent updated application state data can increase the memory footprint of the compressed data, but may be necessary for fast access time because the decompressor must start with the independent application state data and decode all updated application states between the independent context and the target context.

[0047] In addition, through inter-frame prediction, non-reference mutually dependent compressed application state data can be used to speed up decompression. Adding non-reference mutually dependent updated application state data can increase the independent updated application state data interval and / or reduce response delay. Non-reference mutually dependent updated application state data cannot be used to share dictionaries with future contexts. Pages in non-reference mutually dependent updated application state data cannot be used as references for future inter-frame predicted updated application state data. If the decompressor must decode from independent updated application state data to a target updated data state, it can skip the non-reference mutually dependent updated application state data between them to increase decoding speed. For the same example, if the decompressor can decode 50 compressed application states per second and there are two non-reference mutually dependent updated application states between each other context, the independent updated application state data interval can be increased to 75 for the same half-second response delay.

[0048] Timing

[0049] Figure 8is a timing diagram illustrating application state compression in a method for application state compression and replay according to an aspect of the present disclosure. Figure 8 The timing state diagram of shows the interaction of the various elements of the device, each of which includes the main memory (physical main memory), the processor, the mass storage location 1, the mass storage location 2, and the mass storage location 3. It should be noted that for illustrative purposes, three mass storage locations are shown, and a new mass storage location may exist for each instance of compressed application state data or compressed updated application state data. Initially, as indicated at 801, the processor may write application state data 802 to the main memory. The processor may generate application state data by processing an application or by input entered by a user. Once the application state data 802 is written to the main memory, the processor may continue to process the data, and in some implementations, a process switch may occur. The processor application switch may cause the processing to be suspended, and the memory management of the processor may swap the application state data out of the main memory. As indicated at 803, during the swap operation, the processor may copy the application state data 802 from the main memory to the first mass storage location, as indicated at 804. The first mass storage location may be a virtualized main memory storage space. Depending on the implementation, after the swap operation, the application state data may be erased from the main memory and copied back to the main memory (not shown) when the processor begins processing the application again. In addition, if the state memory is not completely swapped out by the process switch, all pages may be forced to be swapped out at 803. After the application state data is copied to the first large capacity storage location, the device may begin compressing the application state data. The processor may compress the copied application state data 804, as indicated by 805. Alternatively, another processing unit may perform the compression, such as, but not limited to, a GPU or a discrete compression processor or a discrete video codec. In an alternative implementation, a GPU and / or a CODEC accelerator may be used for decompression. Once compressed, the application state data is copied to a second location in the large capacity storage (at 806). In some implementations, the second location in the large capacity storage may be a public storage rather than a virtualized main memory.

[0050] After copying the compressed application state data, the device can resume processing the application state data at 807. In some implementations, the processor can resume processing during the compression of the application state data. In these implementations, the compression can be performed by a separate core or thread of the processor or a GPU or a discrete compression processor or a discrete video CODEC. Processing the application state data generates updated application state data 808, which is written to the main memory. Again, the memory management of the processor can initiate a swap operation, where the updated application state data 808 in the main memory is copied to the virtualized main memory space at the first large capacity storage location at 809, as indicated at 810. Once copied to the large capacity storage location, the updated application state data can be compressed by the processor or another discrete processing unit (such as a GPU, a compression processor, or a video codec) at 811. Then, the compressed updated application state data 812 can be written to the third large capacity storage location. Note that the previously compressed application state data is not overwritten, which allows the previous application state data to be used for inter-frame prediction compression or as a different processing starting point. In contrast, during a swap operation, the application state data or updated application state data stored in the virtualized main memory space in the mass storage is erased and overwritten during each swap operation. In addition, each time the compressed application state data or updated application state data is written to the mass storage, it can be written to a new location in the mass storage.

[0051] Figure 9 is a timing diagram depicting application state loading and replay in a method for application state compression and replay according to an aspect of the present disclosure. At some point during processing, the processor may reach a hard pause in processing, as indicated by 902. This hard pause in processing may be determined by the application, or may be selected by the user. For example, but not limited to, the application may be programmed to use a compressed application state (such as a game save) when the application reaches an end state (such as a game end screen). Processing may be paused to allow the compressed application state or the compressed updated application state to be loaded. Alternatively, the user may be provided with the option to pause processing and resume the application from one of the compressed application state or the compressed updated application state. As shown, while processing is paused, the main memory may be filled with the updated application state 901 in the process. The virtualized main memory space at the first mass storage location may be filled with the updated application state data 903 swapped from the main memory. Other locations in the mass storage may be filled with compressed application state data 904 and updated compressed application state data 905.

[0052] After processing is paused, the device may scan the mass storage at 906 to determine the compressed application state 904 and the compressed updated application state 905 stored therein. The device may use the uncompressed tag with the compressed application state data and the compressed updated application state data to determine information about the compressed data 907. Information about the compressed application state 904 and the compressed updated application state 905 may be stored in the main memory 907. In some implementations, the information may include image frames or audio buffers reconstructed from the compressed application state 904 and the compressed updated application state 905. The device may display the information about the compressed application state and the updated application state on a display screen, as indicated by 908. In some implementations, displaying the information about the compressed data may include displaying image frames reconstructed from the compressed data. The original image frames may be stored with the compressed data, and decompression may be applied to the compressed original image frames to reconstruct image frames that can be displayed on the screen. Multiple image frames from the compressed application state data and the compressed updated application state data may be put together to reconstruct a video representation of the compressed storage state. The device can prompt the user to select an application state or an updated application state from the compressed data. Then, the device can wait at 909 until the user selects an application state or an updated application state to be replayed from it.

[0053] Once the processor receives the user's selection of compressed application state data and compressed updated application state data at 912, it can begin loading the selected state into main memory. In order to load the selected compressed application state data 910 or compressed updated application state data 911, the selected state is decompressed at 913. Before or after decompression, the suspended process state data in main memory can (optionally) be erased, as indicated by 914, 915. The selected decompressed application state data 910 or the selected decompressed updated application state data 911 is written to main memory (physical main memory) 917 at 916. Once the selected decompressed data 917 is written to main memory, the processor can start processing 918 from the decompressed data in main memory. Alternatively, the decompressed data can be written to a new location in main memory, and the pointer in the table can be changed to point to the new location in main memory. The decompressed data in main memory represents a previous state during processing, and loading the decompressed data into main memory causes the processor's data to enter that state during processing.

[0054] System

[0055] Figure 101 is a system block diagram of a system implementing a method for application state compression and replay according to an aspect of the present disclosure. As an example and not limitation, according to aspects of the present disclosure, system 1000 may be an embedded system, a mobile phone, a personal computer, a tablet computer, a portable gaming device, a workstation, a gaming console, etc.

[0056] The system 1000 generally includes a central processing unit (CPU) 1002, a graphics processing unit (GPU) 1003, and memory 1004 accessible to both the CPU and the GPU. The system 1000 may also include well-known support functions 1006, which may communicate with other components of the system, for example, via a data bus 1005. Such support functions may include, but are not limited to, input / output (I / O) elements 1007, a power supply (P / S) 1011, a clock (CLK) 1012, and a cache 1013. In addition to the cache 1013, the GPU 1003 may also include its own GPU cache, and the GPU may be configured so that programs running on the GPU 1003 can read or write through the GPU cache.

[0057] The system 1000 may include a display device (not shown) for presenting rendered graphics to a user. In an alternative implementation, the display device is a separate component that works in conjunction with the system 1000. The display device may be in the form of a flat panel display, a head mounted display (HMD), a cathode ray tube (CRT) screen, a projector, or other device that can display visible text, numbers, graphic symbols, or images.

[0058] System 1000 includes a mass storage device 1015, such as a disk drive, CD-ROM drive, flash memory, solid state drive (SSD), tape drive, etc., to provide non-volatile storage for programs and / or data. System 1000 may also optionally include a user interface unit 1016 for facilitating interaction between system 1000 and a user, such as to provide input and control to an application. User interface 1016 may include a keyboard, mouse, joystick, light pen, game controller, or other devices that may be used in conjunction with a graphical user interface (GUI). System 1000 may also include a network interface 1014 for enabling the device to communicate with other devices via a network 1020. Network 1020 may be, for example, a local area network (LAN), a wide area network (such as the Internet), a personal area network (such as a Bluetooth network), or other types of networks. These components may be implemented with hardware, software, or firmware, or some combination of two or more of these.

[0059] The processor 1002 and the graphics processor 1003 may receive and use input from the user interface unit 1016 during application processing, such as information corresponding to keystrokes, button presses, etc. The processor 1002 and the graphics processor 1003 may optionally communicate with a remote client device 1021 via a network 1020 through a network interface 1014. The remote client device may send input, such as information corresponding to keystrokes, button presses, etc. The processor may use these inputs during execution of the application. Image frames and other application data resulting from such execution may be sent to the client device to stream the output of the application running on the processor 1002 and the GPU 1003 to the client device. As a method for Figure 3 , Figure 4 and Figure 9 In the remote host device of the method shown, the processor can receive a processing pause command from the remote client device 1021 and, in response, pause the processing and application state playback. In addition, the processor 1002 can send information about the compressed application state stored in the mass storage 1015 and the updated compressed application state 1017, which can optionally include one or more reconstructed image frames and / or audio buffers from the compressed application state 1017. The processor can receive a selection of the compressed application state in the mass storage 1015 or the compressed updated application state 1017 from the remote client device 1021 to restart processing from.

[0060] The CPU 1002 and the GPU 1003 may each include one or more processor cores, such as a single core, a dual core, a quad core, an octa core, or more. In some implementations, the CPU 1002 and the GPU 1003 may be a core or multiple cores of the same accelerated processing unit (APU) 1001. When the CPU or GPU is a multi-core processor, the underutilized cores may be used for state compression or decompression of stored states. For example, but not limited to, an idle or underutilized core may perform decompression while other cores process other uncompressed data. When applications run in parallel, the underutilized cores may be used for compression.

[0061] The memory 1004 may be in the form of an integrated circuit that provides addressable memory (e.g., random access memory (RAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc.). The memory 1004 may include dedicated graphics memory that may store graphics resources and temporarily store data for a graphics buffer for a graphics rendering pipeline. Figure 10In the example shown in , the graphics memory is shown as part of the application state data 1008 in the main memory 1004. In an alternative implementation, the graphics memory can be a separate hardware component, possibly integrated into the GPU 1003. The main memory 1004 can include application state data or updated application state data 1008 used by the processor 1002 and (optionally) the GPU 1003 when processing. The application state data can be a snapshot of the state of the data being processed by the processor, which has been moved or copied from the register to the main memory, and the updated application state data can be the changed application state data (also called dirty pages) that have not yet been moved from the main memory to the mass storage. Register data is the most volatile data. As a first step to obtain the updated application state data, the register data is copied to the memory 1004. As described above, in the case of using the "dirty page" flag, it is desirable to synchronize the memory swap operation with the application state compression operation. The application state data and the updated application state data 1008 can be stored in memory pages in the memory 1004. During processing, the processor may suspend processing and swap pages of application state data 1008 from main memory 1004 into a virtualized main memory space located in mass storage 1015, so in some implementations, uncompressed application state data or uncompressed updated application state data 1018 may be stored in the virtualized main memory space of mass storage 1015. Although this virtualized main memory space is located on mass storage random access memory 1015, it appears to the processor to be part of the main memory.

[0062] According to various aspects of the present disclosure and as Figure 1 , Figure 2 and Figure 8 As described, during processing, the application state or updated application state 1008 from the main memory can be compressed, and the compressed application state or compressed updated application state 1017 can be stored in a location in the mass storage 1015. In addition, the main memory can include a compression program or algorithm 1009 for use by the processor. In some alternative implementations, the graphics processor 1003 can perform the compression. Or in some other alternative implementations, another discrete processor (such as but not limited to a discrete video CODEC 1023) can perform the compression. The mass storage 1015 may contain application or program data 1022, which is loaded into the main memory 1004 when processing begins on the application or program 1022. As shown in FIG. Figure 5As shown, memory pages in the main memory may include application state data, which includes application context data and is created by the processor during processing of application data 1022. The main memory 1004 may also contain application state selection data 1010, which may include information about compressed application state data 1017 stored in mass storage 1015 and application playback selection and loading, such as Figure 3 , Figure 4 and Figure 9 In some implementations, the application state selection 1010 data may also include one or more reconstructed frames from the compressed application state data or the compressed updated application state data 1017 .

[0063] By way of example and not limitation, CPU 1002 and GPU 1003 may access memory 1004 via one or more buses 1005. In some cases, it may be useful for system 1000 to include two or more different buses. Memory 1004 may contain data that can be accessed by CPU 1002 and GPU 1003. GPU 1003 may include multiple computing units configured to perform graphics processing tasks in parallel. Each computing unit may include its own dedicated local memory storage area, such as a local data share. Alternatively, the computing units may each access memory 1004 or a dedicated graphics memory.

[0064] As used herein and generally understood by those skilled in the art, an application specific integrated circuit (ASIC) is an integrated circuit that is customized for a specific purpose and is not intended for general use.

[0065] As used herein and generally understood by those skilled in the art, a field programmable gate array (FPGA) is an integrated circuit designed to be configured by a customer or designer after manufacture (hence "field programmable"). FPGA configuration is typically specified using a hardware description language (HDL) similar to that used for ASICs.

[0066] As used herein and generally understood by those skilled in the art, a system on a chip or system on a chip (SoC or SOC) is an integrated circuit (IC) that integrates all the components of a computer or other electronic system into a single chip. It can contain digital, analog, mixed signal and usually radio frequency functions - all on a single chip substrate. Typical applications are in the field of embedded systems.

[0067] A typical SoC includes the following hardware components:

[0068] One or more processor cores (e.g., microcontroller, microprocessor, or digital signal processor (DSP) cores.

[0069] Memory blocks such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), and flash memory.

[0070] A timing source, such as an oscillator or a phase-locked loop.

[0071] Peripherals such as counter-timers, real-time timers, or power-on reset generators.

[0072] External interfaces, for example, industry standards such as Universal Serial Bus (USB), FireWire, Ethernet, Universal Asynchronous Receiver / Transmitter (USART), Serial Peripheral Interface (SPI) bus.

[0073] Analog interfaces, including analog-to-digital converters (ADCs) and digital-to-analog converters (DACs).

[0074] Voltage regulators and power management circuits.

[0075] These components are connected by a proprietary or industry standard bus. A direct memory access (DMA) controller routes data directly between the external interface and the memory, bypassing the processor core and thereby increasing the data throughput of the SoC.

[0076] A typical SoC includes both the hardware components described above and executable instructions (eg, software or firmware) that control the processor core, peripherals, and interfaces.

[0077] Aspects of the present disclosure allow for quick access to a previous execution state of an application and restarting the application from a selected previous state. This capability can be applied to video games and other forms of computer simulations, such as flight simulators, driving simulators, and ship navigation simulators.

[0078] Although the above is a complete description of the preferred embodiment of the present invention, various alternatives, modifications and equivalents may be used. Therefore, the scope of the present invention should not be determined by reference to the above description, but should instead be determined by reference to the appended claims and their full range of equivalents. Any feature described herein (whether preferred or not) may be combined with any other feature described herein (whether preferred or not). In the appended claims, Indefinite article "a (A)" or "an (An)" Refers to one or more quantities of the item following the article unless expressly stated otherwise. The following claims should not be interpreted as including means-plus-function limitations unless such a limitation is explicitly recited in a given claim using the phrase "means for..."

Claims

1. A method for application playback, comprising: compressing application state data from main memory; writing the compressed application state data to a first location in mass storage; generating updated application state data; compressing the updated application state data from the main memory; writing the updated application state data to a second location in mass storage; suspending processing of the application state data and the updated application state data; scanning the compressed application state data and the compressed updated application state data stored in the mass storage; Information from the scanned compressed application state data and the compressed updated application state data is used to display information corresponding to the compressed application state data and the updated compressed application state data stored in the mass storage.

2. The method according to claim 1, further comprising: decompressing selected compressed application state data or selected compressed updated application state data stored in the mass storage; writing the decompressed application state data or the decompressed updated application state data to the main memory; The decompressed application state data or the decompressed updated application state data in the main memory is processed. 3 . The method of claim 1 , further comprising reconstructing an image frame from the compressed application state data or the compressed updated application state data. 4 . The method of claim 3 , wherein displaying information corresponding to the compressed application data and updated application data comprises displaying an image frame reconstructed from the compressed application state data or the compressed updated application state data. 5 . The method of claim 1 , further comprising reconstructing an audio buffer from the compressed application state data or the compressed updated application state data. 6 . The method of claim 3 , wherein displaying information corresponding to the compressed application data and updated application data comprises playing an audio buffer reconstructed from the compressed application state data or the compressed updated application state data. 7 . The method of claim 1 , wherein compressing the application state data or compressing the updated application state data further comprises determining a compression type to apply to the application state data or the updated application state data.

8. The method of claim 7, wherein determining the compression type comprises determining whether to apply run-length encoding, dictionary-type compression, or a general compression algorithm.

9. The method of claim 8, wherein the general compression algorithm comprises context tree weighting (CTW), and the dictionary type compression comprises Lemple-Ziv-Welch compression, Lemple-Ziv-Huffman-Arithmetic-Markov (LZHAM) compression, or DEFLATE compression.

10. The method of claim 7, wherein compressing the updated application state data comprises determining whether to apply inter-frame prediction to the updated application state data. 11 . The method of claim 10 , wherein applying inter-prediction to the updated application state data comprises performing an exclusive-OR between the application state data and the updated application state data.

12. The method of claim 11, wherein applying inter-frame prediction to the updated application state data further comprises dictionary compressing the inter-frame prediction and sharing a compression dictionary with the compressed application state data.

13. The method of claim 1, wherein generating the updated application state data comprises processing two or more image frames, and compressing the updated application state data comprises compressing the updated application state data after processing the two or more image frames. 14 . The method of claim 13 , wherein compressing the updated application state data comprises compressing the updated application state data after processing a set interval of image frames.

15. The method of claim 14, wherein the image frames at set intervals are defined by a user or the application state data.

16. The method of claim 1, wherein the application state data and updated application state data include an image frame and an application context.

17. The method of claim 16, wherein the application context includes processor registers, a stack, and a heap. The method of claim 16 , wherein the application state data further comprises simulation device state data.

19. The method of claim 1, wherein displaying information corresponding to the compressed application state data and updated compressed application state data stored in the mass storage further comprises sending the information to a remote client.

20. The method of claim 19, wherein the processing is suspended in response to a request from a remote client.

21. The method of claim 2, further comprising clearing suspended in-process application state data or in-process updated application state data from the main memory before writing the decompressed application state data or the decompressed updated application state data to the main memory.

22. The method of claim 2, wherein processing the decompressed application state data or the decompressed updated application state data in main memory comprises restarting the application at a point in application processing that generated the application state data or the updated application state data prior to compression.

23. The method of claim 1 further comprising copying the application state data from a physical main memory to a virtual main memory space before compressing the application state data, and comprising copying the updated application state data from the physical main memory to the virtual main memory space before compressing the updated application state data.

24. The method of claim 23, wherein the virtual main memory space comprises a portion of non-volatile memory reserved for the virtual main memory space.

25. A system for application playback, comprising: processor; a main memory coupled to the processor; a mass storage coupled to the main memory and the processor; Instructions, embodied in the main memory, which when executed by the processor cause the processor to perform a method for application replay, the method comprising; compressing application state data from the main memory; writing the compressed application state data to a first location in the mass storage; generating updated application state data; compressing the updated application state data from the main memory; writing updated application state data to a second location in the mass storage; suspending processing of the application state data and the updated application state data; scanning the compressed application state data and the compressed updated application state data stored in the mass storage; Information from the scanned compressed application state data and the compressed updated application state data is used to display information corresponding to the compressed application state data and the updated compressed application state data stored in the mass storage.

26. A non-transitory computer readable medium having executable instructions embodied thereon, the executable instructions, when executed by a computer, causing the computer to perform a method for application playback, the method comprising: compressing application state data from main memory; writing the compressed application state data to a first location in mass storage; generating updated application state data; compressing the updated application state data from the main memory; writing the updated application state data to a second location in mass storage; suspending processing of the application state data and the updated application state data; scanning the compressed application state data and the compressed updated application state data stored in the mass storage; Information from the scanned compressed application state data and the compressed updated application state data is used to display information corresponding to the compressed application state data and the updated compressed application state data stored in the mass storage.

Images