Token stream differencing with moved-block detection

Abstract

Methods and apparatus implementing systems and techniques for differencing token streams and detecting moved blocks of tokens. In general, in one implementation, the technique includes: obtaining a first token stream and a second token stream, comparing the first and second token streams to identify a group of tokens that are substantially similar in the first and second token streams, the similar-tokens group including common sub-sequences, which are identical in the first and second token streams, and at least one unmatched token, and presenting matched token information corresponding to the similar-tokens group to represent changes in document flow.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS[0001]This application is a continuation application of and claims priority to U.S. application Ser. No. 10 / 272,858 filed on Oct. 16, 2002. The disclosure of the prior application is considered part of (and is incorporated by reference in) the disclosure of this application.BACKGROUND OF THE INVENTION[0002]The present application describes systems and techniques relating to token stream differencing, for example, comparison of text documents to identify document changes.[0003]Various techniques exist for comparing token streams. Such comparison is commonly referred to as differencing or as a diff operation. Differencing two token streams typically involves comparing two versions of a token stream, commonly referred to as the original stream and the modified stream, and looking for differences between them. In the context of text comparison, many differencing processes use individual text characters or words as the tokens. Such diff processes ar...

AI Technical Summary

Benefits of technology

[0006]In general, in one aspect, the invention involves obtaining a first token stream and a second token stream, comparing the first and second token streams to identify a group of tokens that are substantially similar in the first and second token streams, the similar-tokens group including common sub-sequences, which are identical in the first and second token streams, and at least one unmatched token, and presenting matched token information corresponding to the similar-tokens group to represent changes in document flow. Implementations of the invention can include one or more of the following advantageous features.

[0014]The invention can be implemented to realize one or more of the following advantages. Move detection, including detection of moved blocks that were changed slightly but are still substantially similar, allows simpler change tracking between revisions of a document. Additions and deletions are also identified, and the techniques can be used with tokens that represent various types of sequential data. In text documents, a word-level token granularity can be used, while still effectively detecting moved blocks of text with small modifications relative to the size of the block. Moves that occur within a single line can be detected, and text reflows need not cause problems for moved-block detection.

[0015]Very good moved-block detection performance can be obtained, even on token streams that have many similarities and are of roughly the same length. A predefined sequences differencing technique can be used as a replaceable component with the systems and techniques of the invention. This design modularity enables additional performance improvements to be realized in the future when faster differencing techniques become available.

[0016]Moreover, moved blocks of tokens can be grouped together before a final presentation. Collecting the identified moved blocks into larger groups can reduce visual complexity of a final results presentation, and the grouped blocks can be displayed using colors to assist a reviewer. For example, when the tokens are words in a text document, displaying the grouped blocks using multiple colors can assist a reader in identifying changes in document flow.

Problems solved by technology

Moreover, when such techniques actually do identify moved blocks, the displayed results can be very confusing because small additions and / or deletions within a moved block of text can create a checker-boarding effect in the generated results, where moved and unmoved words interleave each other, thus cluttering the results report.

Images