Compression and replay of application process context

Abstract

The present invention provides a method, system, and medium that enables rapid access to a previous execution state of an application and restart of the application from a selected previous state. [Solution] The method involves writing compressed application state data and updated application state data from main memory to first and second locations in the large-capacity storage, respectively, pausing processing for the application state data and updated application state data, scanning the compressed application state data and compressed updated application state data stored in the large-capacity storage, and displaying information corresponding to the compressed application state data and updated compressed application state data stored in the large-capacity storage using the information from the scanned compressed application state data and compressed updated application state data.

Description

Technical Field

[0001] Aspects of the present disclosure relate to the management of memory, and more particularly to application replay from compressed main memory pages.

Background Art

[0002] Modern computer architectures use main memory to store data that the processor operates on. The processor can transfer information from the main memory to its registers, execute computations, and then transfer the contents of the registers back to the main memory. Such main memory contains application state data, which is a snapshot of the state of the data being processed by the processor that has been moved from the registers to the main memory. Application state data can include an application context such as processor registers, stacks, and heaps. Main memory data is periodically saved to non-volatile memory, also known as mass storage, such as a hard disk drive (HDD) or a solid state drive (SSD). During processing, the processor can update the application state data in the main memory without updating the data stored in the mass storage, and since the page data is updated without being written to the mass storage, this is sometimes referred to as a dirty page. Application state data is stored in one or more pages in the main memory, and each page is composed of a plurality of fixed-size page frames or variable-size page segments. The processor maintains an address buffer that holds the location of the application state data within the main memory.

[0003] Many operating systems implement memory swapping, which allows them to increase the size of main memory without requiring additional physical memory. Memory swapping uses a portion of non-volatile memory storage space as main memory. Swapping retrieves application state data from main memory and writes it to non-volatile memory space while application processing is paused. When the operating system changes the application being processed, it may swap the application state data written to this area of ​​non-volatile memory back and forth between the physical memory of main memory and the non-volatile memory area. Swapping pages from physical memory to non-volatile storage, and restoring pages from non-volatile memory to physical memory, are controlled by two different algorithms. If an application attempts to allocate a page of memory but physical memory is full, the operating system pauses the application and selects a page that is not actively being used to copy to non-volatile memory. The operating system then reclaims that page for memory allocation and resumes the application. Another reason for swapping out pages is switching application contexts. When one application switches out, its memory pages are also swapped out to make room for the application being swapped in.

[0004] It has been proposed that memory management efficiency can be improved by compressing application data stored in a portion of main memory. Various compression techniques have been proposed to reduce the size of application state data stored in main memory. However, these techniques are not yet widely adopted across the industry.

[0005] Video games and other applications save application state data using independent points programmed within the application. These are often referred to as "game saves," "state saves," or "save files." Depending on the embodiment, these save files may be periodically updated or created by the game application or at the user's discretion. The creation of save data operates independently of the processor's memory management and creates save data in mass storage, regardless of main memory or memory swapping. The content of save data is variable depending on the embodiment, and is not standardized across applications.

[0006] In this situation, the nature of this disclosure arises. [Overview of the project]

[0007] The teachings in this disclosure can be easily understood by considering the following detailed description in conjunction with the attached drawings. [Brief explanation of the drawing]

[0008] [Figure 1] This is a timing diagram showing application state compression according to the aspects of this disclosure. [Figure 2] This is a flowchart illustrating application state compression in a method for compressing and replaying application state according to an aspect of the present disclosure. [Figure 3] This is a flowchart illustrating application state selection in a method for compressing and replaying application states according to an aspect of this disclosure. [Figure 4] This is a flowchart illustrating application state replay loading in a method for compressing and replaying application state according to an aspect of the present disclosure. [Figure 5] A block diagram showing the contents of a memory page, including the application state, according to the aspects of this disclosure. [Figure 6]This is a flowchart illustrating application state compression in a method for compressing and replaying application state according to an aspect of the present disclosure. [Figure 7] This is a flowchart illustrating updated application state compression in a method for compressing and replaying application state according to an aspect of the present disclosure. [Figure 8] This is a timing diagram showing application state compression in a method for compressing and replaying application state according to an aspect of the present disclosure. [Figure 9] This is a timing diagram showing application state loading and replay in a method for compressing and replaying application state according to an aspect of the present disclosure. [Figure 10] This is a block schematic diagram of a system implementing a method for compressing and replaying application state according to an aspect of the present disclosure. [Modes for carrying out the invention]

[0009] The following detailed description includes many specific details for illustrative purposes, but those skilled in the art will recognize that many variations and modifications of the following details fall within the scope of the invention. Accordingly, exemplary embodiments of the invention described below are shown without loss of generality to the claimed invention and without imposing limitations on the claimed invention.

[0010] While it has been shown that page compression can improve memory management efficiency, these techniques have not been widely adopted due to a lack of hardware resources. Currently, high-speed multi-core processors are common, and hardware resources are abundant enough to implement page compression without impacting processing. Memory management systems and other applications still do not utilize memory compression.

[0011] Previously proposed memory compression methods involve directly compressing memory pages into main memory during processing, but according to aspects of this disclosure, higher system functionality can be achieved by compressing and storing the compressed memory pages in mass storage. Storing compressed memory pages in mass storage enables application replay, allowing users to return to past points in the application's processing. For example, but not limited to, storing compressed memory pages in mass storage may allow playback of a video game from a previous point in a video game that was not saved, resuming a simulator program at a point between simulations where the state was not saved, reverting changes made in a drawing or writing program where the previous state was not saved, reviewing the stored compressed application state to determine the application's context and find bugs during program debugging, and recovering lost application data after a system crash using the stored compressed application state. The advantages of storing compressed application state are numerous and allow for the addition of functionality to programs that previously lacked the ability to save state or did not have frequent save points. Furthermore, the compressed application state contains more information than the saved files created by the application, potentially enabling more granular application replays or more user-friendly application replay points.

[0012] method Figure 1 shows an embodiment of application state compression according to an aspect of the present disclosure. In this particular embodiment, an application, for example, a game application, performs processing over a predetermined time interval and generates output in the form of data corresponding to an image frame at the end of each interval. However, the aspects of the present disclosure are not limited to such embodiments. As shown, initially, the system can perform processing and generate a first image frame 101. After the generation of the first image frame, the system pauses at 102, and the application state data, paged in main memory, is copied at 103 to a virtualized main memory space in a first location of mass storage. The application state data in main memory pages is then compressed at 104 and copied at 105 to a second location of mass storage. As used herein and as generally understood by those skilled in the art, “state” is a general term representing all the information necessary to define a system. The “context” of an application refers more specifically to all the information necessary to restore a computer application.

[0013] The processor resumes processing the application state data, and a second image frame 106 is generated. The generation of the second image frame 106 may include generating updated application state data to be stored in main memory. In some embodiments, the application process may be paused at 107, and the updated application state data generated in the second frame is copied from physical main memory to virtual main memory in a first location of mass storage 108. The application state data generated in the second frame may then be compressed at 109 and copied to a third location of mass storage 110.

[0014] In an alternative embodiment, one or more image frames may be processed with the corresponding updated application state data or application data without compressing the memory pages of the updated application state data. Thus, elements 107-110 are indicated by dashed lines to show that the operation is optional depending on the embodiment. In another alternative embodiment, the application state data copied in 103 and the updated application state data copied in 108 may also be stored in physical memory. The copied pages corresponding to the state data are snapshots of the application. Memory compression may be performed on snapshots of physical memory.

[0015] As shown, a third image frame 111 may be generated with updated application state data. Next, processing may be paused at 112, and the memory page corresponding to the updated application state data may be copied to the virtual main memory space at a first location in mass storage at 113. Then, the updated application state data generated in the third image frame may be compressed at 114 and copied to a fourth location in mass storage. Alternatively, if the previous updated application state data was not compressed, the compressed updated application state data may be copied to a third location in mass storage. This process may continue up to the nth image 115, where n is the number of image frames generated while the application is running.

[0016] As shown in Figure 1, in some embodiments, the pause during processing may only last long enough to copy the application state data from main memory to virtual main memory space. According to some aspects of this disclosure, the application state data can be compressed and copied to mass storage outside of the swap operation using parallel processing with a processor (e.g., a second thread, core, or processing unit), a GPU, a discrete codec, etc. In some embodiments, a processor other than the CPU, such as a GPU, may be used to decompress the compressed or compressed updated application state data.

[0017] Furthermore, it should be noted that in some cases, processed application state data may be stored in mass storage independently of the compressed paging method of this disclosure. For example, as described above, the operating system may swap some inactive pages from physical memory. If these pages are "clean," i.e., unchanged from the state of the previous interval or frame, they can be compressed and stored directly without expanding the pages into physical memory. However, if a page is "dirty," i.e., changed from the state of the previous interval or frame, the latest memory content from physical memory must be selected and stored. In such situations, the "clean page" and "dirty page" flags may be used as indicators to assist the memory compressor. If a page can remain clean after one processing interval, e.g., one frame iteration, the compressor only needs to remove this page from the current frame context and use an index to reference the page in the context of the previous frame. Otherwise, the stored dirty page is stored along with the context of the current interval, such as the current frame. When using the "dirty page" flag, it is desirable to synchronize page memory swapping with the operation of the application state compressor.

[0018] If the application state compressor is running with virtual memory management, memory page swapping may occur synchronously with application state compression. In other words, the operating system only performs page memory swapping when the application compressor needs that page memory for compression. In this case, the dirty page flag means that the page has been changed between the application state data and the updated application state data. Alternatively, if the operating system swaps memory pages independently of the application state compressor, dirty pages are only those pages that have been changed since the last page swap, but the timing of the last swap is independent of application state data compression and updated application state compression. In such situations, the dirty page flag is not useful for determining whether the data on the flagged page has been changed, and the application compressor should always assume that the data has been changed.

[0019] As described above, aspects of the present disclosure are not limited to only the embodiments shown in FIG. 1. FIG. 2 more generally shows details of application state compression according to aspects of the present disclosure. A method for application state compression and replay can begin 201 with the selection of an application. The processor may pull application data from mass storage to main memory for processing. Next, the processor can process the application and generate application state data. After pausing the processing, the application state data is copied 202 from the registers of the processor to a page in main memory as shown. In some optional embodiments, the processor may perform a swap operation by copying the application state data from physical main memory to a first location, such as a virtual memory space in mass storage, or a location in main memory at 203. Next, as shown at 204, the processor reads and compresses the application state data from the first location. In some alternative embodiments, the swap operation may be skipped and instead the processor can read and compress the application state data directly from main memory at 204, so the swap operation is not necessary for compression of the application state. Next, the compressed page of application state data is written 205 to a second location in mass storage. This second location in mass storage is outside the virtualized main memory space, so the processor does not operate as if it were main memory in this region of mass storage. After pausing the processing, the processor can resume processing the application state data and process one or more image frames from the application state data to create updated application state data stored 206 as a page in physical main memory.

[0020] As before, the processor pauses, and the updated application state data can be (optionally) swapped from the pages of the physical main memory to the virtual main memory space of the first location of the mass storage at 207. Next, the updated application data is compressed at 208 and written to the third location of the mass storage at 209. Note that the compression of the updated application state data need not be performed after each processing interval, for example after every full image frame, but can occur at intervals such as every 2 frames, every 3 frames, every 5 frames, or every n frames, where the number of frames n is determined by the application, the operating system, or based on the available resources. Alternatively, n may be determined by the user.

[0021] The processor can continue to process the application state data and generate image frames and updated application state data at 210, or the application can end the processing and compression of the application state as shown at 211. The end state can occur when the processing ends or when the program is interrupted. Alternatively, the end state can be an application processing pause selected by the user to return to the previous application state or the updated application state.

[0022] In many use cases, it is desirable to synchronize the page swap - out with the application state compression. Instead of swapping out pages when the physical memory is full or switching applications, the pages can be periodically copied from one location to another for application state compression.

[0023] Furthermore, since the swapped - out memory pages are compressed before being stored, the standard operating system process can be changed to store the swapped - out pages in another location of the physical memory instead of non - volatile memory. Usually, the data compressor operates more easily in physical memory. Only the compressed data is stored in non - volatile memory.

[0024] As described above, memory page swapping can be performed synchronously with or independently of application state compression. Clean and dirty page flags can be used in conjunction with such embodiments, as described above.

[0025] Figure 3 is a block diagram showing application state selection in a method for compressing and replaying application state according to an aspect of the present disclosure. First, the processor may move from the application state compression method to the application state selection method due to an interruption in application processing, such as a programmed interruption or application failure, or due to user input. The processor pauses processing of the current application state data, as shown in 301. Next, the system may scan the mass storage device for the compressed and updated application state data in 302. In some optional embodiments, the system may reconstruct image frames associated with one or more states from the application state data or updated application state data 303. Furthermore, in some other alternative embodiments, audio associated with one or more states from the application state data may be reconstructed from the audio buffer of the application state data or updated application state data 303. The audio may be played back, for example, through a speaker, and may be played back with or without the display of image frames. According to aspects of this disclosure, some embodiments of application state data and updated application state data may include associated image frames and / or audio buffers that are compressed using the application state data or updated application state data. Decompression of the compressed image frames and / or audio buffers within the application state data or updated application state data can reconstruct the image frames for display or the audio for playback. Alternatively, the application context can be processed to reconstruct the image frames or audio buffers.Using information from scanned compressed application state data and compressed updated application state data, the system can display or play information corresponding to the compressed application state data and compressed updated application state data in mass storage 304. In such an optional embodiment, this information may include reconstructed image frames or audio buffers generated from the compressed or compressed updated application state data.

[0026] In some embodiments, a list of compressed application state data and compressed updated application state data may be displayed, and the list may be arranged chronologically. To facilitate the rapid chronological arrangement of the display, each instance of compressed application state data and compressed updated application state data may include a tag describing attributes such as the date and time of creation and the application instance. The display of information corresponding to the compressed application data and compressed updated application data may be generated, for example, by displaying the information in the tags on a display screen. In some embodiments, this may also include the display of reconstructed image frames and / or playback of audio buffers reconstructed from the compressed data. Once the information corresponding to the compressed application state data and compressed updated application state data is displayed, the processor 305 awaits a selection by the user. In some alternative embodiments, image frame data from the compressed application state data or compressed updated application state data may be displayed upon selection by the user 306. In some embodiments, audio buffer data from the compressed application state data or compressed updated application state data may be played upon selection by the user 306, with or without the display of image frames. This embodiment can reduce unnecessary deployment cycles and provide an intuitive representation of the selected state of the application while still providing functionality to the user. The deployed image frames and / or audio buffers can be reconstructed before the user confirms that it is desirable to resume the application from the selected compressed or compressed updated application state. As the compressed or compressed updated application state is selected or confirmed to be selected by the user, the system may start the application using the compressed or compressed updated application state data, as described below, for example, with respect to Figure 4, in 307.

[0027] The flowchart shown in Figure 4 illustrates application state replay loading in a method for compressing and replaying application state according to an aspect of the present disclosure. After selecting a compressed or compressed updated application state in 401, the system initializes processing from the compressed or compressed updated application state. To begin, the compressed or compressed updated application state is decompressed as shown in 402. In some embodiments, the compressed or compressed updated application state data is copied to memory, where a processor performs an operation to decompress the data. In alternative embodiments, a separate processing unit, such as a graphics processor, or discrete compression / decompression circuitry, such as a video encoder / decoder (CODEC), may perform the decompression and copy the compressed or compressed updated application state data to a memory space reserved for the separate processing unit. To decompress compressed or updated application state data, the system may use the reverse of the algorithm used to compress the data to decompress the data, which may include using the reverse operation of a common compression algorithm, or applying X-ORd data to a base dataset of mutual prediction data and using the same dictionary used for dictionary encoding. Before, during, or after decompression, in some optional embodiments, the application state data in main memory where processing is suspended may be cleared with a 403 to make room for the decompressed or decompressed updated application state data. Alternatively, a portion of the application state data in main memory where processing is suspended may be retained, and one or more pointers to the corresponding memory location(s) may be changed to the location(s) of the decompressed or decompressed updated application state data when it is copied to main memory.Compressed or compressed and updated application state data, once decompressed, can be copied to main memory in 404. As described, the decompressed and compressed application state data, or the decompressed and compressed and updated application state data, can be copied to the same location as the application state data or updated application state data that was in the previous process. Alternatively, as discussed above, the decompressed and compressed application state data, or the decompressed and compressed and updated application state data, can be copied to a new location in main memory, and the pointer(s) to a different table (such as the translation lookaside buffer) may be changed. Application processing can be resumed in the same way as normal context switching, for example, from one application to another. Specifically, the normal context switching process may include a process of loading registers from memory into the processor before the processor is started. Such a loading process, sometimes called initialization, may be performed before processing is resumed. Finally, once the decompressed and compressed application state data, or the decompressed and compressed and updated application state data, is copied to main memory and any necessary initialization is performed, in 405, the processor can be unsuspended and begin processing the state data copied in main memory.

[0028] Therefore, in practice, according to several aspects of this disclosure, a gamer can use captured gameplay video (reconstructed from compressed application state data and compressed updated application state data) to search for replay points. The game video can be played forward or backward. If the gamer finds a suitable replay point, they can pause the video and start the replay at that point. To enable such use cases, a set of timecodes can be embedded as tags in both the compressed application state data and the compressed updated application state data. If the gamer pauses the video at a particular frame, the timecode of the video frame can be passed to the decompression hardware and / or software to retrieve the application state or updated application state for that timecode. Note that the frame rate of the game video may differ from the rate at which the application context is captured. A one-to-one match of timecodes between the captured context and the captured video may not exist.

[0029] compression The block diagram shown in Figure 5 illustrates an example of the contents of a memory page containing application state according to an aspect of this disclosure. As shown, memory page 501 consists of a plurality of fixed-size frames 502. In some alternative embodiments, a memory page may consist of a plurality of variable-size segments, the size of which may be recorded in a table. Application state data and updated application state data 507 may extend to multiple frames or segments of memory page 501. In some embodiments, application state data and updated application state data may include both image frame data and / or audio buffer data 503 and associated application context data 504. Alternatively, application state data or updated application state data may be application context data without associated image frames or audio buffers. Note that image frames are data representing pixel values ​​for display on a display screen. Audio buffers are data representing samples of audio signals for playback by speakers or other sound-producing devices. Application context data may include application data, additional main processor and GPU registers, GPU memory and local memory of the GPU compute unit, the stack, and the heap. A stack is a linear data structure that tracks variables used by the processor during processing in an ordered manner. A heap is a large, unordered memory space for other variables and memory objects created during processing. Memory pages can also contain other data, such as data for other applications or operating system processes. Although memory pages are shown as being separated into frames, memory pages can be partitioned into smaller memory blocks. A larger compressed memory block size can improve compression efficiency, while a smaller block size can result in faster compression. The compressor can determine an appropriate compression model for each block. Statistically, larger block sizes result in better compression.In some embodiments, the compressed memory block may be of variable size, using tags or tables that indicate the characteristics of the memory block, such as size and location, but not limited to these. In some embodiments, the compressed and updated application state data may include decompressed tags that describe attributes of the data, such as creation time, application, and size. The tags may also indicate which parts of the compressed data are image frames or audio buffers. The tags may be compressed or uncompressed. Uncompressed tags are faster to retrieve. Compressed tags offer better compression efficiency.

[0030] The flowchart shown in Figure 6 illustrates an example of application state compression in a method for compressing and replaying application state according to aspects of the present disclosure. First, application state data can be modeled to find available regularities within the data in a given page, block, segment, or other data unit, as shown in 601. Modeling may determine regularities such as the number of zeros in the data or repetitions of specific data strings or structures. After modeling, the device can determine the best compression algorithm by applying different compression algorithms 605, 606, and 603 and measuring the compression of the resulting compressed application state data 602. Compression may be measured using the final memory size of the compressed application state data. The memory sizes of the compressed application state data created by general compression 605, lexicographical compression 606, and run-length compression 603 can be compared in 604 to determine the compressed application state data with the smallest memory size. The smallest compressed application state data can then be stored in mass storage 607. In some alternative embodiments, the compression ratio may be obtained using the final compressed file size. The compression ratio may be the ratio of the memory size of the compressed application state data to the memory size of the uncompressed application state data. Compression ratios can be generated for the general compression algorithm 605, the dictionary compression algorithm 606, and the run-length encoding algorithm 603. All ratios of the applied compression algorithms can be compared in 604, and the algorithm that yields the smallest memory size can be selected and stored in the mass storage 607.

[0031] If modeling and initial compression determine that a unit of application state data in question contains many zero- or single-digit repetitions, run-length coding may be applied to that unit of data stored in mass storage. Run-length coding takes each single-digit repetition and reduces it from a state of many single-digit repetitions to a two-entry coding that presents the digit and the number of repetitions in order. For example, but not limited to, an array like AAAAABBBBCCDDDDD coded with run-length coding becomes A5B4C2D5, and thus the 16-character array is reduced to 8.

[0032] The modeling data and initial compression may determine whether there are any available regularities in the data based on the size or compression ratio of the compressed application state data. If the application state data contains available regularities, lexicographic coding, such as Remple-Ziv-Welsh Family (LZW) compression, DEFLATE compression, or Remple-Ziv-Huffman-Arithmetic-Markkoff (LZHAM) compression, may be applied to the compressed application state data stored in mass storage. Lexicographic coding can compress the application state data by replacing the regularities in the application state data with tokens and placing the tokenized regularities in a table (dictionary). If the regularities in the data are insufficient to be coded by the lexicographic compression method, as determined by the comparison of the compressed memory size 604 from the modeling 601 and the initial application of the compression algorithm, the final compressed application state data stored in mass storage 607 may be compressed with another common compression algorithm, such as, for example, Context Tree Weighting (CTW) coding or other similar lossless compression algorithms, but are not limited to these.

[0033] Figure 7 is a block diagram showing updated application state compression in a method for compressing and replaying application state according to an aspect of the present disclosure. First, the memory pages of the updated application state data are modeled to determine regularity and similarity between the updated application state data and previously generated application state data, as shown in 701. After modeling, the best compression algorithm can be determined by applying compression algorithms such as mutual prediction in general compression 708, mutual prediction in lexicographical compression 707, mutual prediction in run-length coding 705, and a novel compression algorithm 703 as seen in Figure 6. Compression can be measured using the final memory size of the compressed application state data 702 produced by each algorithm 708, 707, 705, and 703. The memory sizes of the compressed application state data created by each algorithm 708, 707, 705, and 703 can be compared in 704 to determine the compressed application state data with the smallest memory size. The compressed application state data with the smallest memory size can then be stored in mass storage in 709. In some alternative embodiments, the compression ratio may be obtained using the final compressed file size. The compression ratio may be the ratio of the memory size of the compressed application state data to the memory size of the uncompressed application state data. Compression ratios may be generated for each of the compression types 708, 707, 705, and 703. All ratios of the applied compression algorithms may be compared in 704, and the algorithm that yields the smallest memory size may be selected and stored in the mass store 709. The device can use the data modeling and the memory size of the compressed state data to determine whether there is any similarity in the updated application state data to any previous application state data. If there is no similarity, the device may compress the updated application state data using a new compression algorithm, i.e., a non-mutual-predictive algorithm as described above and shown in Figure 6.If similarity exists between the updated application state data and the previous application state data, as determined by modeling 701 and the application of mutual prediction and compression, the device uses data modeling and compressed data size to determine that the data contains many zero or single-digit repetitions. If many single-digit or zero repetitions are found, mutual prediction using run-length coding may be applied to the updated application state data. Mutual prediction involves taking the exclusive OR (XOR) of the updated application state data and the previous application state data to create a page of data containing only the difference between the updated application state data and the previous application state data. Run-length coding may be applied to the page of modified data created by the XOR operation and stored in the mass store 709, based on the comparison of compressed memory sizes in 704. Next, the run-length coding described above removes single-digit repeating strings containing two entry encoders. If the number of single-digit repetitions is insufficient, the device can determine, through the application of a compression algorithm and the comparison in 704, that there is regularity in the data and that lexicographical compression may be used for the stored compressed state data. If there is a sufficient number of regularities in the updated application state data, as determined by the application of the compression algorithm and comparison 704, the stored updated application state data may be compressed, for example, in LZW, using mutual prediction by lexicographical coding. When paired with mutual prediction, lexicographical coding can use the same coding table (dictionary) used for the previous application state data.

[0034] If there is not enough regularity in the updated application state data, mutual prediction with a general encoding method may be applied to the stored compressed state data. As mentioned above, the general encoding method may be, for example, the lossless compression method of CTW, but is not limited to this. In some embodiments, there may be a lot of updated application state data with few "clean pages" of application state data. Therefore, to enable random access, independent updated application state data may be inserted and compressed as application state data (shown in Figure 6) at intervals based on the decompression speed. For example, if 50 compressed updated application states can be decompressed per second, compressed independent updated application state data may be created for every 25 compressed updated application states. In this example embodiment, a maximum response time of 0.5 seconds occurs. The response time may be faster if the selected updated application state data is close to independent updated application state data. Independent updated application state data includes all memory pages, not just dirty memory pages. Furthermore, if independent updated application state data is encoded according to the method shown in Figure 6, mutual prediction is not applied to independent updated application state data. Inserting independently updated application state data may increase the memory footprint of compressed data, but it may require fast access times because the decompressor must start with the independent application state data and decode all updated application state between the independent context and the target context.

[0035] Furthermore, mutual prediction can accelerate decompression by using non-referenced interdependent compressed application state data. By adding non-referenced interdependent updated application state data, the interval between independently updated application state data can be increased and / or response delays can be reduced. In the future, it will no longer be possible to share context and dictionaries using non-referenced interdependent updated application state data. Pages of non-referenced interdependent updated application state data cannot be used as references to future mutual prediction updated application state data. If the decompressor has to decode from independently updated application state data to the target updated data state, non-referenced interdependent updated application state data between them can be skipped to improve decoding speed. For the same example, if the decompressor can decode 50 compressed application states per second, and there are two non-referenced interdependent updated application states between any other contexts, the interval between independently updated application state data can be increased to 75 with the same response delay of 0.5 seconds.

[0036] timing Figure 8 is a timing diagram illustrating application state compression in a method for compressing and replaying application state according to an aspect of the present disclosure. The timing diagram of Figure 8 shows the interaction of device elements, including main memory (physical main memory), the processor, mass storage location 1, mass storage location 2, and mass storage location 3. Three mass storage locations are shown for illustrative purposes, and it should be noted that there may be a new mass storage location for each instance of compressed or compressed updated application state data. Initially, as shown in 801, the processor can write application state data 802 to main memory. The processor can generate application state data by processing an application or by input entered by the user. Once the application state data 802 is written to main memory, the processor can continue processing the data, and in some embodiments, a process switch may occur. A processor application switch may cause a pause in processing, and the processor's memory management may swap the application state data from main memory. As shown in 803, during a swap operation, the processor may copy application state data 802 from main memory to a first mass storage location, as shown in 804. The first mass storage location may be a virtualized main memory storage space. Depending on the embodiment after the swap operation, when the processor starts processing the application again, the application state data may be erased from main memory and copied back to main memory (not shown). Furthermore, if the state memory is not completely swapped out by the process switch, all pages may be forcibly swapped out in 803. After the application state data has been copied to the first mass storage location, the device may begin compressing the application state data. The processor may compress the copied application state data 804, as shown in 805.Alternatively, another processing unit, for example, but not limited to, a GPU or a discrete compression processor or discrete video CODEC, can perform the compression. In alternative embodiments, a GPU and / or CODEC accelerator may be used for decompression. Once compressed, the application state data is copied to a second location of 806 mass storage. In some embodiments, the second location of mass storage may be common storage instead of virtualized main memory.

[0037] After copying the compressed application state data, the device can resume processing the application state data at 807. In some embodiments, the processor can resume processing while the application state data is being compressed. In these embodiments, compression may be performed by a separate core or thread of the processor, or by a GPU, or by a discrete compression processor or discrete video codec. Processing the application state data generates updated application state data 808, which is written to main memory. In this case as well, the processor's memory management can initiate a swap operation at 809, where the updated application state data 808 from main memory is copied at 810 to a virtualized main memory space at a first mass storage location. Once copied to the mass storage location, the updated application state data may be compressed at 811 by the processor, or by another separate processing unit such as a GPU, compression processor, or video codec. The compressed updated application state data 812 may then be written to a third mass storage location. Note that the previously compressed application state data was not overwritten. This allows previous application state data to be used for mutual predictive compression or as a starting point for different processing. In contrast, during a swap operation, application state data or updated application state data held in the virtualized main memory space of mass storage is erased and overwritten during each swap operation. Furthermore, each time compressed or updated application state data is written to mass storage, it may be written to a new location on mass storage.

[0038] Figure 9 is a timing diagram showing application state loading and replay in a method for compressing and replaying application state according to an aspect of the present disclosure. At some point during processing, the processor may reach a hardware pause in processing, as shown in 902. This hardware pause in processing may be determined by the application or selected by the user. For example, without limitation, an application may be programmed to use a compressed application state, such as a game save, when the application reaches an exit state, such as a game overscreen. Processing can be paused to allow loading of a compressed or compressed updated application state. Alternatively, the user may be given the option to pause processing and resume the application from one of the compressed or compressed updated application states. As shown, while processing is paused, an in-process updated application state 901 may be populated into main memory. The updated application state data, swapped from main memory, may be populated in 903 into a virtualized main memory space located in a first mass storage location. Other locations in the large-capacity storage may contain compressed application state data 904 and updated compressed application state data 905.

[0039] After processing is paused, the device can scan the mass store at 906 to determine the compressed application state 904 and the compressed updated application state 905 stored therein. The device can use the decompressed tags along with the compressed application state data and the compressed updated application state data to determine information 907 about the compressed data. Information about the compressed application state 904 and the compressed updated application state 905 may be stored in main memory at 907. In some embodiments, this information may include image frames or audio buffers reconstructed from the compressed application state 904 and the compressed updated application state 905. The device can display information about the compressed application state and the updated application state on a display screen, as shown in 908. In some embodiments, displaying information about the compressed data may include displaying image frames reconstructed from the compressed data. Raw image frames may be stored along with the compressed data, and decompression may be applied to the compressed raw image frames to reconstruct image frames that can be displayed on the screen. Multiple image frames from compressed application state data and compressed updated application state data can be combined to reconstruct a video representation of the compressed stored state. The device can prompt the user to select an application state or an updated application state from the compressed data. The device can then wait at 909 until the user has selected an application state or an updated application state and started replaying.

[0040] When the processor receives a user selection from compressed and compressed updated application state data at 912, the processor can begin loading the selected state into main memory. The selected state is decompressed at 913 in order to load the selected compressed application state data 910 or compressed updated application state data 911. Before or after decompression, suspended in-process state data in main memory may be (optionally) erased as shown in 914, 915. The selected decompressed application state data 910 or selected decompressed updated application state data 911 is written to main memory (physical main memory) 917 at 916. Once the selected decompressed data 917 is written to main memory, the processor can begin processing 918 from the decompressed data in main memory. Alternatively, the decompressed data may be written to a new location in main memory, and the table pointer may be changed to point to the new location in main memory. The decompressed data in main memory represents the previous state during processing, and when the decompressed data is loaded into main memory, the processor's data is placed in that state during processing.

[0041] system Figure 10 is a block system diagram of a system implementing a method for compressing and replaying an application state according to one aspect of the present disclosure. Without limitation, but as an example, according to aspects of the present disclosure, system 1000 may be an embedded system, a mobile phone, a personal computer, a tablet computer, a portable game device, a workstation, a game console, and the like.

[0042] System 1000 generally includes a central processing unit (CPU) 1002, a graphics processing unit (GPU) 1003, and memory 1004 accessible to both the CPU and the GPU. System 1000 may also include well-known support functions 1006 that can communicate with other components of the system, for example, via a data bus 1005. Such support functions may include, but are not limited to, input / output (I / O) elements 1007, a power supply (P / S) 1011, a clock (CLK) 1012, and a cache 1013. In addition to the cache 1013, the GPU 1003 may include its own GPU cache, and the GPU may be configured to allow programs running on the GPU 1003 to read through or write through the GPU cache.

[0043] System 1000 may include a display device (not shown) for presenting rendered graphics to the user. In an alternative embodiment, the display device is a separate component that functions in conjunction with System 1000. The display device may be in the form of a flat panel display, a head-mounted display (HMD), a cathode ray tube (CRT) screen, a projector, or other device capable of displaying visible text, numbers, graphic symbols, or images.

[0044] Furthermore, system 1000 includes a mass storage device 1015, such as a disk drive, CD-ROM drive, flash memory, solid-state drive (SSD), tape drive, etc., to provide non-volatile storage for programs and / or data. System 1000 may also optionally include a user interface unit 1016 to facilitate interaction between system 1000 and the user, for example, to provide input and control to an application. The user interface 1016 may include a keyboard, mouse, joystick, light pen, game controller, or other devices that can be used with a graphical user interface (GUI). System 1000 may also include a network interface 1014 to enable the device to communicate with other devices through a network 1020. The network 1020 may be, for example, a local area network (LAN), a wide area network such as the Internet, a personal area network such as a Bluetooth® network, or other types of networks. These components may be implemented in hardware, software, or firmware, or any combination of two or more of these.

[0045] Processor 1002 and graphics processor 1003 may receive and use information corresponding to input from the user interface unit 1016, such as keystrokes and button presses, while processing the application. Optionally, processor 1002 and graphics processor 1003 may communicate with a remote client device 1021 via the network interface 1014 through the network 1020. The remote client device may transmit input such as information corresponding to keystrokes and button presses. The processors may use these inputs while executing the application. Image frames and other application data resulting from such execution may be sent to the client device, and the application output executed by processor 1002 and GPU 1003 may be streamed to the client device. A processor acting as a remote host device in the manner shown in Figures 3, 4, and 9 receives a processing pause command from the remote client device 1021, pauses processing in response, and replays the application state. Furthermore, the processor 1002 can transmit information about the compressed application state and the updated compressed application state 1017 stored in the mass storage 1015, which may optionally include one or more reconstructed image frames and / or audio buffers from the compressed application state 1017. The processor may receive a selection of the compressed application state or the compressed updated application state 1017 from the remote client device 1021 in the mass storage 1015 to resume processing.

[0046] The CPU 1002 and GPU 1003 may each include one or more processor cores, for example, a single core, two cores, four cores, eight cores, or more. In some embodiments, the CPU 1002 and GPU 1003 may include multiple cores of the same Accelerator Unit (APU) 1001. If the CPU or GPU is a multi-core processor, underutilized cores may be used for state compression or decompression of stored state. For example, but not limited to, if other cores are processing other uncompressed data, a free or underutilized core may perform decompression. Underutilized cores may be used for compression while applications are running in parallel.

[0047] Memory 1004 may be in the form of an integrated circuit that provides addressable memory, such as random access memory (RAM), dynamic random access memory (DRAM), or synchronous dynamic random access memory (SDRAM). Memory 1004 may include dedicated graphics memory that can store graphics resources and temporarily store graphics buffers of data for the graphics rendering pipeline. In the example shown in Figure 10, graphics memory is shown as part of the application state data 1008 of main memory 1004. In alternative embodiments, graphics memory may be a separate hardware component and may be integrated into the GPU 1003. Main memory 1004 may include application state data or updated application state data 1008 used by the processor 1002 and (optionally) the GPU 1003 during processing. Application state data may be a snapshot of the state of data being processed by the processor that has been moved or copied from registers to main memory, and updated application state data may be modified application state data that has not yet been moved from main memory to mass storage, also known as dirty pages. Register data is the most volatile data. As the first step in obtaining the updated application state data, the register data is copied to memory 1004. As mentioned above, when the "dirty page" flag is used, it is desirable to synchronize the memory swap operation with the application state compression operation. The application state data and the updated application state data 1008 can be stored in the memory pages of memory 1004.During processing, the processor can pause processing and swap pages of application state data 1008 from main memory 1004 to a virtualized main memory space located in mass storage 1015. In some embodiments, uncompressed application state data, or uncompressed updated application state data 1018, can be stored in the virtualized main memory space of mass storage 1015. This virtualized main memory space appears to the processor as part of main memory, even though it is located in mass storage random access memory 1015.

[0048] According to aspects of this disclosure, the application state or updated application state 1008 from main memory may be compressed during processing, as discussed in Figures 1, 2, and 8, and the compressed or compressed updated application state 1017 may be stored in a location in mass storage 1015. Furthermore, main memory may include a compression program or algorithm 1009 for use by the processor. In some alternative embodiments, the graphics processor 1003 may perform the compression. Alternatively, in some other alternative embodiments, another discrete processor, for example, a discrete video CODEC 1023, may perform the compression. Mass storage 1015 may include application or program data 1022 that is loaded into main memory 1004 when processing is initiated by the application or program 1022. As shown in Figure 5, memory pages in main memory may include application context data and application state data created by the processor during processing of the application data 1022. The main memory 1004 may also include compressed application state data 1017 stored in the mass storage 1015 as shown in Figures 3, 4, and 9, and application state selection data 1010 which may include information regarding application replay selection and loading. In some embodiments, the application state selection data 1010 may also include one or more frames reconstructed from the compressed application state data or compressed updated application state data 1017.

[0049] As an example, and not an limitation, the CPU 1002 and GPU 1003 may access memory 1004 via one or more buses 1005. In some cases, it may be useful for system 1000 to include two or more different buses. Memory 1004 may contain data that the CPU 1002 and GPU 1003 can access. GPU 1003 may include multiple compute units configured to execute graphics processing tasks in parallel. Each compute unit may include its own dedicated local memory store, such as a local data share. Alternatively, each compute unit may access memory 1004 or dedicated graphics memory.

[0050] As used herein and as generally understood by those skilled in the art, an application-specific integrated circuit (ASIC) is an integrated circuit customized for a specific application, rather than for general-purpose use.

[0051] As used herein and as generally understood by those skilled in the art, a field-programmable gate array (FPGA) is an integrated circuit designed to be configured by the customer or designer after manufacture—and thus "field-programmable." FPGA configurations are generally specified using a hardware description language (HDL) similar to that used for ASICs.

[0052] As used herein and as generally understood by those skilled in the art, a system on a chip, or system-on-a-chip (SoC or SOC), is an integrated circuit (IC) that integrates all the components of a computer or other electronic system onto a single chip. It can include digital, analog, mixed-signal, and often radio frequency functions—all on a single chip substrate. Typical applications are in the field of embedded systems.

[0053] A typical SoC includes the following hardware components: One or more processor cores (e.g., a microcontroller, microprocessor, or digital signal processor (DSP) core). Memory blocks, such as read-only memory (ROM), random-access memory (RAM), electrically erasable programmable read-only memory (EEPROM), and flash memory. Timing source, e.g., oscillator or phase-locked loop. Peripheral devices, such as counter timers, real-time timers, or power-on reset generators. External interfaces, such as industry standards including Universal Serial Bus (USB), FireWire®, Ethernet, USART (universal asynchronous receiver / transmitter), and Serial Peripheral Interface (SPI) bus. Analog interface (including analog-to-digital converters (ADCs) and digital-to-analog converters (DACs)). Voltage regulator and power management circuit.

[0054] These components are connected by either a proprietary bus or an industry-standard bus. The Direct Memory Access (DMA) controller improves the SoC's data throughput by routing data directly between the external interface and memory, bypassing the processor core.

[0055] A typical SoC includes both the hardware components mentioned above, as well as the processor core(s), peripherals, and executable instructions (e.g., software or firmware) that control the interfaces.

[0056] Aspects of this disclosure enable rapid access to previous execution states of an application and restart of the application from a selected previous state. Such functionality has applications to video games, as well as other forms of computer simulations, such as flight simulators, driving simulators, and ship navigation simulators.

[0057] The above is a complete description of preferred embodiments of the present invention, but various alternatives, modifications, and equivalents are possible. Therefore, the scope of the present invention should not be determined by reference to the above description, but rather by reference to the appended claims, according to the full extent of those equivalents. Any feature described herein, whether preferred or not, may be combined with any other feature described herein, whether preferred or not. In the following claims, the indefinite article "A" or "An" refers to one or more items following the article, unless explicitly stated otherwise. The appended claims should not be construed as including means-plus-function limitations unless such limitations are explicitly stated in the given claim using the phrase "means for".

Claims

1. The computing system compresses application state data from main memory. The computing system writes the compressed application state data to a first location in the mass storage. The computing system generates application state data that is updated at least partially based on the application state data. The computing system compresses the updated application state data from the main memory. The computing system writes the compressed and updated application state data to a second location in the mass storage, and A method comprising the computing system reconstructing an image frame from the compressed updated application state data from the main memory.

2. The further includes deciding to update the aforementioned application status data based on an indicator, The method according to claim 1, wherein the compressed and updated application state data is written using a context based on the indicator.

3. The compressed and updated application state data includes audio information. The method according to claim 1, further comprising reconstructing the audio buffer from the compressed updated application state data based on the audio information.

4. To display an image based on the compressed and updated application state data, Receiving input indicating the selection of the aforementioned image, and The method according to claim 1, further comprising causing the application associated with the compressed updated application state data to run based on the input.

5. The compressed and updated application state data is written to the allocated memory space, and The method according to claim 1, further comprising decompressing the compressed updated application state data from the allocated memory space.

6. To generate a model that represents the aforementioned application state data and shows at least one of the number of zeros in the application state data and the repetition of the data string of the application state data, and The method further includes selecting a compression algorithm based on at least one of the number of zeros in the application state data or the repetition of the data string in the application state data. The method according to claim 1, wherein the application state data is compressed using the selected compression algorithm.

7. To generate a model that represents the aforementioned application state data and shows the similarity between the aforementioned application state data and the updated application state data, and The method further includes selecting a compression algorithm based on the similarity between the application state data and the updated application state data. The method according to claim 1, wherein the application state data is compressed using the selected compression algorithm.

8. A computing system, One or more processors, and Includes one or more computer-readable media on which instructions are stored, When the aforementioned instruction is executed, the instruction shall Compressing application state data from main memory, The compressed application state data is written to a first location in the mass storage. To generate application state data that is updated at least partially based on the aforementioned application state data, Compress the updated application state data from the main memory. The compressed and updated application state data is written to the second location of the large-capacity storage, and A computing system that causes one or more processors to reconstruct image frames from the compressed and updated application state data from the main memory.

9. When the aforementioned instruction is executed, the instruction shall Further cause one or more processors to decide to update the application state data based on the indicator, The computing system according to claim 8, wherein the compressed and updated application state data is written using a context based on the indicator.

10. The compressed and updated application state data includes audio information. When the aforementioned instruction is executed, the instruction shall The computing system according to claim 8, further comprising causing one or more processors to reconstruct an audio buffer from the compressed and updated application state data based on the audio information.

11. When the aforementioned instruction is executed, the instruction shall To display an image based on the compressed and updated application state data, Receiving input indicating the selection of the aforementioned image, and The computing system according to claim 8, further comprising causing one or more processors to execute the application associated with the compressed and updated application state data based on the input.

12. When the aforementioned instruction is executed, the instruction shall The compressed and updated application state data is written to the allocated memory space, and The computing system according to claim 8, further comprising causing one or more processors to decompress the compressed updated application state data from the allocated memory space.

13. When the aforementioned instruction is executed, the instruction shall To generate a model that represents the aforementioned application state data and shows at least one of the number of zeros in the application state data and the repetition of the data string of the application state data, and The one or more processors are further made to select a compression algorithm based on at least one of the number of zeros in the application state data or the repetition of the data string in the application state data. The computing system according to claim 8, wherein the application state data is compressed using the selected compression algorithm.

14. When the aforementioned instruction is executed, the instruction shall To generate a model that represents the aforementioned application state data and shows the similarity between the aforementioned application state data and the updated application state data, and The one or more processors are further instructed to select a compression algorithm based on the similarity between the application state data and the updated application state data. The computing system according to claim 8, wherein the application state data is compressed using the selected compression algorithm.

15. One or more non-temporary computer-readable media on which instructions are stored, When the aforementioned instruction is executed, the instruction shall Compressing application state data from main memory, The compressed application state data is written to a first location in the mass storage. To generate application state data that is updated at least partially based on the aforementioned application state data, Compress the updated application state data from the main memory. The compressed and updated application state data is written to the second location of the large-capacity storage, and A non-temporary, computer-readable medium that causes one or more processors to reconstruct image frames from the compressed, updated application state data from the main memory.

16. When the aforementioned instruction is executed, the instruction shall Further cause one or more processors to decide to update the application state data based on the indicator, The non-temporary computer-readable medium according to claim 15, wherein the compressed updated application state data is written using a context based on the indicator.

17. The compressed and updated application state data includes audio information. When the aforementioned instruction is executed, the instruction shall A non-temporary computer-readable medium according to claim 15, further comprising causing one or more processors to reconstruct an audio buffer from the compressed updated application state data based on the audio information.

18. When the aforementioned instruction is executed, the instruction shall To display an image based on the compressed and updated application state data, Receiving input indicating the selection of the aforementioned image, and A non-temporary computer-readable medium according to claim 15, further causing one or more processors to execute an application associated with the compressed updated application state data based on the input.

19. When the aforementioned instruction is executed, the instruction shall The compressed and updated application state data is written to the allocated memory space, and A non-temporary computer-readable medium according to claim 15, further comprising causing one or more processors to decompress the compressed updated application state data from the allocated memory space.

20. When the aforementioned instruction is executed, the instruction shall To generate a model that represents the aforementioned application state data and shows at least one of the number of zeros in the application state data and the repetition of the data string of the application state data, and The one or more processors are further made to select a compression algorithm based on at least one of the number of zeros in the application state data or the repetition of the data string in the application state data. The non-temporary computer-readable medium according to claim 15, wherein the application state data is compressed using the selected compression algorithm.

Images