Method and device for identifying at least one field in bits of a bit stream

Abstract

A method of identifying at least one field in bits of a bit stream is disclosed. The method includes obtaining multiple first bit streams of a same type and obtaining a number of bit flips for bits in each bit position of multiple first bit streams. The method further includes identifying adjacent bits of a first bit stream that do not yet belong to an identified field as belonging to a NUMERIC field if the corresponding number of bit flips are in a descending order from a least significant bit position to a most significant bit position of the adjacent bits and the adjacent bits are of at least a predetermined bit length. A device including a processor configured to perform the method is also disclosed.

Description

METHOD AND DEVICE FOR IDENTIFYING AT LEAST ONE FIELD IN BITS OF A BIT STREAMTECHNICAL FIELD

[0001] This invention relates to a method and a device for identifying at least one field in bits of a bit stream. More particularly, this invention relates to a method and a device for identifying at least one field in bits of a payload in a message.BACKGROUND

[0002] The following discussion of the background to the invention is intended to facilitate an understanding of the present invention only. It should be appreciated that the discussion is not an acknowledgement or admission that any of the material referred to was published, known or part of the common general knowledge of the person skilled in the art in any jurisdiction as at the priority date of the invention.

[0003] Protocol Reverse Engineering (PRE) is required to understand communications that occur within a telecommunications network. More specifically, it is required to understand what exchanged messages represent when protocol specifications are not readily available, especially in the case involving proprietary protocols. To that end, PRE can infer the fields of messages to determine the message format.

[0004] Manual PRE methods are costly and require experienced analysts with sufficient domain knowledge of specific protocols and their underlying mechanisms to be able to do the job. To mitigate this problem, automatic PRE methods are known to have been implemented. One such method is disclosed in Brent C. Stone, "Enabling auditing and intrusion detection of proprietary controller area networks". The article discloses an unsupervised CAN payload reverse engineering approach for non-text network protocols that groups and labels signals inferred. Its multi-staged pipeline groups payloads by their arbitration ID before further analysis. Lexical analysis is used to separate data into its logically unique tokens for identification while semantic analysis attempts to find meaning of each token that was identified. Through both approaches, correlations and causal relationships between the tokens are learnt. The semantic analysis focuses on only numeric signals but not other types of fields. And byte order in a numeric field is only determined for individual signals. Identifyingwhether a NUMERIC field is signed or unsigned is carried out by observing the frequency of bit flips of the LSB and MSB. The frequency at which values change from a positive to a negative value implies a NUMERIC field to be signed. Such an approach is not very reliable.

[0001] There is therefore a need for a method of identifying fields in a bit stream which addresses, at least in part, one or more of the forgoing problems.SUMMARY

[0002] According to an aspect of the present disclosure, there is provided a method of identifying at least one field in bits of a bit stream. The method includes obtaining multiple first bit streams of a same type and obtaining a number of bit flips for bits in each bit position of multiple first bit streams. The method further includes identifying adjacent bits of a first bit stream that do not yet belong to an identified field as belonging to a NUMERIC field if the corresponding number of bit flips are in a descending order from a least significant bit position to a most significant bit position of the adjacent bits and the adjacent bits are of at least a predetermined bit length.

[0003] In some embodiments of the method, the method further includes identifying a bit of the first bit stream as belonging to a STATIC field if the corresponding number of bit flips is zero prior to identifying adjacent bits of the first bit stream as belonging to the NUMERIC field.

[0004] In some embodiments of the method, the method further includes identifying a bit of the first bit stream that do not yet belong to an identified field as belonging to a FLAG field if bitflips occur over one bit for that bit.

[0005] In some embodiments of the method, the method further includes obtaining multiple second bit streams of a same type as the first bit streams. The second bit stream has a different bit length than the first bit stream. The method further includes obtaining a number of bit flips for bits in each bit position of the plurality of second bit streams, identifying STATIC, FLAG and NUMERIC fields to which bits of a second bit stream belong in a same manner as that for the first bit stream based on the number of bit flips for bits in each bit position of the multiple second bit streams, and identifying adjacent bits of the first bit stream and the second bit stream that do not yet belong to an identified field as belonging to a VARIABLE field by comparing the identified fields of the first bit stream and the second bit stream.

[0006] In some embodiments of the method, comparing the identified fields of the first bit stream and the second bit stream includes matching identified fields in the first bit stream and the second bit stream to locate boundaries of the adjacent bits in the first bit stream and the second bit stream so as to identify them as belonging to the VARIABLE field.

[0007] In some embodiments of the method, the adjacent bits in the first bit stream and the adjacent bits in the second bit stream that are identified as belonging to the VARIABLE field are of unequal bit lengths.

[0008] In some embodiments of the method, identifying a bit of a STATIC field to be a most significant bit of a NUMERIC field immediately adjacent to the STATIC field if the corresponding number of bit flips of the bit is zero.

[0009] In some embodiments of the method, identifying a bit to be a least significant bit of a NUMERIC field immediately adjacent to the bit if the corresponding number of bit flips of the bit is within a predetermined range.

[0010] In some embodiments of the method, the first bit stream includes two bytes or more bytes and identifying bits of the first bit stream as belonging to a STATIC field and a NUMERIC field are performed with the two or more bytes arranged in a little endian format and a big endian format. The method further includes determining if the two or more bytes are arranged in the little endian format or the big endian format based on a number of fields identified in the first bit stream, a sum of the number of bit flips for the most significant bit of each identified field and a number of bits that do not belong to any identified field for the two or more bytes arranged in both the little endian format and the big endian format.

[0011] In some embodiments of the method, determining if the first bit stream is of one of the little endian format and the big endian format is based on a first difference between the number of fields identified, a second difference between the sum of the number of bit flips for the most significant bit of each field and a third difference between the number of bits that do not belong to any identified field for the at least two bytes in the little endian format and the big endian format.

[0012] In some embodiments of the method, the method further includes determining if the bits of a NUMERIC field are of a signed or unsigned format based on a difference in value of the bits in adjacent pairs of the first bit streams in the plurality of first bit streams.

[0013] According to another aspect of the present disclosure, there is provided a device having a processor that is configured to perform any of the above-described methods.

[0014] According to yet another aspect of the present disclosure, there is provided a program storage device readable by a computing device, tangibly embodying a program of instructions, executable by the computing device to perform any of the above-described methods.

[0015] Other aspects and advantages of the invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention.BRIEF DESCRIPTION OF DRAWINGS

[0016] The invention will be better understood with reference to the drawings, in which:Figure 1 is a flowchart showing a sequence of steps in a method for identifying one or more fields in bits of a bit stream;Figure 2 is a table showing multiple occurrences of the bit stream in Figure 1 ;Figure 3 is a table showing information related to bit flips in bits of the multiple occurrences of the bit stream in Figure 2;Figure 4 shows a pseudo code for identifying fields in the bit stream based on the bit flips in Figure 3;Figure 5 shows field type bitmaps for illustrating how a variable field can be identified;Figure 6 illustrates how boundaries of an identified numeric field may be extended;Figure 7 shows multiple occurrences of two bytes of the bit stream in Figure 1 arranged in a Big Endian order;Figure 8 shows multiple occurrences of the same two bytes in Figure 7 arranged in a Little Endian order;Figures 9 and 10 illustrate how to determine if the two bytes of the bit stream is arranged in a Big Endian or Little Endian format;Figure 11 is a table showing unsigned bit values in multiple occurrences of a numeric field, the corresponding unsigned values and the difference in unsigned values between pairs of adjacent numeric field occurrences;Figure 12 is a table showing signed bit values in multiple occurrences of a numeric field, the corresponding unsigned values and the difference in unsigned values between pairs of adjacent numeric field occurrences; andFigure 13 is a block diagram illustrating typical elements of a computing system that may be appropriately programmed to perform the method in Figure 1 .DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] Throughout this document, unless otherwise indicated to the contrary, the terms “comprising”, “consisting of”, “having” and the like, are to be construed as non- exhaustive, or in other words, as meaning “including, but not limited to.”

[0018] Furthermore, throughout the specification, unless the context requires otherwise, the word “include” or variations such as “includes” or “including” will be understood to imply the inclusion of a stated integer or group of integers but not the exclusion of any other integer or group of integers.

[0019] Throughout the description, it is to be appreciated that the term ‘processor / controller’ and its plural form include microcontrollers, microprocessors, programmable integrated circuit chips such as application specific integrated circuit chip (ASIC), computer servers, electronic devices, and / or combination thereof capable of processing one or more input electronic signals to produce one or more output electronic signals.

[0020] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as is commonly understood by a skilled person to which the subject matter herein belongs.

[0021] As shown in the drawings for purposes of illustration, the invention may be embodied in a method of identifying one or more fields in bits of a bit stream. Existing <method tends to be focused on only identifying a numeric field. Referring to Figures 1 to 3, the method generally includes obtaining multiple first bit streams of a same type and obtaining a number of bit flips for bits in each bit position of multiple first bit streams. The method further includes identifying adjacent bits of a first bit stream that do not yet belong to an identified field as belonging to a NUMERIC field if thecorresponding number of bit flips are in a descending order from a least significant bit position to a most significant bit position of the adjacent bits and the adjacent bits are of at least a predetermined bit length.

[0022] Specifically, Figure 1 shows steps in a sequence 2 for identifying fields of bits in a bit stream according to an embodiment of the invention. The sequence 2 starts in a CAPTURE MESSAGES step 4, wherein a controller area network (CAN) logger (not shown) is used to capture CAN messages 6 on a CAN communications channel. Each CAN message includes a CAN ID. The CAN logger typically outputs the CAN messages 6 in a hexadecimal format for further processing by a device 8 (Figure 13). The device 8 may be the CAN logger itself or a separate device that receives the CAN messages 6 output by the CAN logger. The device 8 may be any computing device including but not limited to a server.

[0023] The sequence 2 next proceeds to a GROUP MESSAGES BY TYPE step 10, wherein the device 8 groups the CAN messages 6 by type. For example, the CAN messages 6 in this embodiment are grouped by their CAN IDs. In Figure 1 , TYPE A messages 12 may be CAN messages with a CAN ID of 1 , TYPE B messages 14 may be CAN messages with a CAN ID of 2, TYPE C messages 16 may be CAN messages with a CAN ID of 3, etc. The sequence 2 next proceeds to a CONVERT TO BINARY step 18, wherein the device 8 converts a payload of each CAN message 6 in hexadecimal format to a bit stream (also known as binary sequence) which is a sequence of bits. In Figure 1 , binary A will be bit streams 20 obtained from the payloads of the TYPE A messages 12. Binary B will be bit streams 22 obtained from the payloads of TYPE B messages 14. And likewise, binary C will be bit streams 24 obtained from the payloads of TYPE C messages 16. Hereinafter, the description of the sequence 2 will be based on only bit streams of a single group or cluster of CAN messages. There may be three or more bit streams in the cluster. Each bit stream may have any bit length. Figure 2 shows a Boolean matrix 26 including eight bit streams 28 in time order in the group with each bit stream 28 having a bit length of one byte (eight bits) with the most significant bit (27bit) shown on the extreme left and the least significant bit (2° bit) shown on the extreme right. In this case, the bit streams 28 are time series data. However, the bit streams may also be any numbered series data. The bit streams 28 will be referred to hereinafter as first bit streams or occurrences of afirst bit stream 28 type. It is to be appreciated that the sequence 2 may also be used on bit streams in the other groups of CAN messages.

[0024] The sequence 2 next proceeds to an OBTAIN NUMBER OF BIT FLIPS step 30, wherein the device 8 obtains the number of bit flips over time for bits in each bit position of the cluster of first bit streams 28 in the Boolean matrix 26. In other words, the device 8 will obtain the number of bit flips over time for a column or segment 32 of bits in the 27position, another segment 32 of bits in the 26position and so on. To determine if there is a bit flip between two bits in each segment 32, the device 8 applies an exclusive-OR bit operation on the two bits. For all the bits in each segment 32 in the Boolean matrix 26 in Figure 2, the device 8 performs an exclusive-OR operation on each pair of adjacent bits in that segment 32. A result of a logical 1 in an exclusive-OR operation indicates that there is a bit flip. In other words, the bit has changed from a logical 0 to a logical 1 , or changed from a logical 1 to a logical 0. For the eight bits in a segment 32, seven exclusive-OR operations are performed on the seven pairs of adjacent bits. For the segment 32 of bits in the 27bit position in Figure 2, there is only one single change in the value of the bit between the first bit stream #4 and the first bit stream #5. For the segment 32 of bits in the 23bit position, there is a change in the value of the bit between a first bit stream 28 and a next first bit stream 28. Figure 3 shows a transition aggregation matrix 34 including the results of the exclusive-OR operations on the first bit streams 28 in the Boolean matrix 26. The number of 1 ’s obtained for each segment 32 of bits are added to give the number of times that particular bit has flipped over the course of the eight first bit streams 28. For the segment 32 of bits in the 27bit position in Figure 2, the number of bit flips over the course of the eight first bit streams is 1 as shown in Figure 3. For the segment 32 of bits in the 23bit position in Figure 2, the number of bit flips over the course of the eight first bit streams 28 is 7 as shown in Figure 3. The number of bit flips for the bits in a first bit stream 28 type is referred to as a transition aggregation N-gram (TANG) vector 36. The TANG vector 36 includes eight values corresponding to the eight bits in the first bit stream 28 type. Each bit in the first bit stream type therefore has a corresponding or associated TANG vector value indicating the number of bit flips for that bit over the course of the eight first bit streams 28.

[0025] The sequence 2 next proceeds to an IDENTIFY FIELDS step 38, wherein the device 8 identifies a field to which each bit in a first bit stream belongs based onthe TANG vector 36 values. Field types include, but are not limited to, a STATIC (S) field type, a VOID (U) field type, a FLAG (F) field type, a NUMERIC (N) field type, and a VARIABLE (V) field type. A bit whose value does not change over the course of the eight first bit streams 28 is identified by the device 8 as belonging to or representing a STATIC field in the first bit stream 28. Such a bit is identified by a corresponding value of zero in the TANG vector 36. Adjacent bits in the first bit stream 28 are identified as belonging to or representing a NUMERIC field when there is a decrease in the corresponding values in the TANG vector 36 from that corresponding to a bit in a least significant bit position to that corresponding to a bit in a most significant bit position. Typically, a bit immediately adjacent those of a first NUMERIC field is of a different field type or of a second NUMERIC field if the corresponding TANG vector 36 value changes drastically from that of an adjacent bit in the first NUMERIC field. A bit with bit flips that is isolated, i.e. occurring over the span of just one bit, is identified as belonging to or representing a FLAG field. Adjacent bits in the first bit stream 28 belonging to or representing a VARIABLE field of a variable bit length are identified to be those that are bounded at both ends by identified fields having fixed lengths. These fields having fixed lengths define the boundaries of the variable length field. Bits whose field type has yet to be identified, i.e. unidentified, or cannot be identified due to there being insufficient information are associated with a VOID field.

[0026] Figure 4 shows an example of pseudo code by which the device 8 may identify the fields in a first bit stream based on the TANG vector 36 values. The device 8 first allocates a data structure, referred to as “bitmap” 40 in Figures 3 and 4, to store information of the fields to which the bits of the first bit stream 28 belong. Each value in the bitmap 40 therefore indicates the field to which the corresponding bit in the first bit stream 28 belongs. The device 8 next loops through each value in the TANG vector 36. If any of the TANG vector 36 values is found to be equal to zero, the device sets the corresponding value in the bitmap 40 to indicate that the corresponding bit belongs to or represents a STATIC field. If a value is not equal to zero, the corresponding field is set to be of the VOID field type in the bitmap 40. For the TANG vector 36 shown in Figure 3, only the bitmap value corresponding to the bit in bit position 26is set as a STATIC field. The remaining values in the bitmap 40 are initially set to the VOID field type. The device 8 next sets a most significant bit (MSB) index and a least significant bit (LSB) index to be none / nulL The device 8 loops through thevalues in the bitmap 40 and TANG vector 36 using a pointer. If it is found that a current value in the bitmap 40 pointed to by the pointer is equal to the VOID field type, the device 8 sets the MSB index and LSB index to the pointer value if both the MSB index and the LSB index are null. If it is found that a current corresponding TANG value is greater than the next TANG value, the device 8 sets the MSB index to the pointer value + 1 .

[0027] If it is determined by the device 8 that the current corresponding TANG value is not greater than the next TANG value, the device 8 obtains a difference in the values between the MSB index and the LSB index. If it is determined that the difference in the values between the MSB index and the LSB index is equal to 0, the device 8 sets the value in the bitmap 40 pointed to by the LSB index to a FLAG field. If, however, it is found that the difference in the values between the MSB index and the LSB index is not equal to 0, the device 8 further checks if the difference between the MSB index and the LSB index is greater than or equal to a predetermined bit length. If it is determined that the difference between the MSB index and the LSB index is greater than or equal to the predetermined bit length, the device 8 sets the values in the bitmap 40 by the MSB index and the LSB index to the NUMERIC field type. The device 8 also resets the MSB index and the LSB index to null. It is to be noted that according to the sequence 2, the bitmap 40 values having a corresponding TANG vector 36 value of zero are set to the STATIC field type prior to the device 8 setting the other bitmap 40 values to the NUMERIC field type.

[0028] For the TANG vector 36 shown in Figure 3, the bitmap 40 will have the values (U S U U U U U U) set after the device 8 has executed lines 2-8 of the pseudo code in Figure 4. These bitmap 40 values will be updated to (U S U U U N N N) after the device 8 has executed lines 9-24 of the pseudo code based on the bitmap 40 values and the TANG vector 36 values corresponding to the three least significant bits of the first bit stream 28, where the device 8 would have determined those three adjacent bits as having a decreasing number of bit flips from a least significant bit position to a most significant bit position thereof and the adjacent bits are of at least the predetermined bit length and therefore identified them as belonging to or representing a first NUMERIC field 42. Similarly, the next three TANG vector 36 values corresponding to the three bits in the bit positions 23to 25will be processed in a similar manner such that these three bits are also identified as belonging to or representing asecond NUMERIC field 44. And the bitmap 40 values will be updated to (U S N N N N N N), leaving the value corresponding to the bit in the 27position to be the only one of the VOID field type. When processing this last TANG vector 36 value, the device 8 identifies the corresponding bit as belonging to or representing a FLAG field since the TANG vector value is non-zero and are bounded by bits of other identified fields, i.e., the bit is an isolated one with bit flips occurring over one bit. In other words, bitflips for that bit are independent of those of adjacent bits. At the end of executing the pseudo code, the bitmap 40 will have the values (F S N N N N N N) as shown in Figure 3.

[0029] Although the first bit streams 28 shown in Figure 2 are all of the same bit lengths, there may be other bit streams (not shown) of the same type as the first bit streams 28 but of a different bit length. For example, there may be second bit streams of the same type but of a shorter bit length than the first bit streams 28. The device 8 proceeds to obtain a corresponding TANG vector (not shown) for these second bit streams in a similar manner as that described above and use it to identify the fields to which the bits in the second bit streams belong or represent. The fields in the second bit streams are identified by the device 8 executing the same pseudo code shown in Figure 4 that is described above and will not be repeated. The steps for identifying VARIABLE fields in a first bit stream 28 and a second bit stream is next described with the aid of Figure 5. Figure 5 shows a first bitmap 40 having values different from that described above and a second bitmap 50 containing field values to which respective bits in the first bit stream 28 and the second bit stream have been identified to belong to. The device 8 compares the two bitmaps 40, 50 to identify any VARIABLE field therein. The device 8 does so by using fields 52, 54 in the bitmaps 40, 50 that have been identified with a high level of confidence. The device 8 uses these fields 52 with a high confidence level to identify boundaries of any VARIABLE field in each of the first bitmap 40 and the second bitmap 50. This may be carried out by left aligning both bitmaps 40, 50 and taking the bit position to the right of the fields 52 that are identified with a high confidence level in each bitmap 40, 50 to be a MSB boundary (pointed to by arrow A) of a respective VARIABLE field. Similarly, the device 8 right aligns both bitmaps 40, 50 and takes the position to the left of fields 54 that are identified with a high confidence level in each bitmap 40, 50 to be a LSB boundary (pointed to by arrow B) of the respective VARIABLE field. The values in each bitmap 40, 50 between the MSB boundary and the LSB boundary are then set to a VARIABLE field type toindicate that the corresponding bits in the bit streams belong to a respective VARIABLE field 56, 58. In the first bitmap 40 in Figure 5, the VARIABLE field 56 has a length of four bits. And in the second bitmap 50 in Figure 5, the VARIABLE field 58 has a length of two bits. In this particular case, the two VARIABLE fields 56, 58 have different bit lengths.

[0030] The sequence 2 next proceeds to an EXTEND NUMERIC FIELD step 60, wherein for each NUMERIC field that is identified in the IDENTIFY FIELDS step 38 described above, the device 8 determines if it is possible to extend the boundaries of an identified NUMERIC field 62 towards the byte boundaries based on the values of the TANG vector 36 and the bitmap 40. The device 8 checks the bitmap value and the TANG vector value corresponding to a bit next to the MSB of the NUMERIC field 62. If it is determined that the bitmap value corresponding to the next bit in the first bit stream is the STATIC field type and the value of each of the corresponding bits of the first bit streams is zero, the MSB of the NUMERIC field 62 is extended to the next bit. This is repeated to further extend the MSB of the NUMERIC field until the bitmap value corresponding to the next bit is not the STATIC field type or any one value of the corresponding bits of the first bit streams is not equal to zero. For the bitmap 40 shown in Figure 6, the MSB boundary of the NUMERIC field 62 is extended all the way to the byte boundary.

[0031] The device 8 also checks the bitmap 40 value and the TANG vector 36 value corresponding to a bit next to the LSB of the NUMERIC field 62. If it is determined that the difference in the TANG vector 36 values corresponding to LSB of the NUMERIC field 62 and the next bit to the right is within a predetermined range, the LSB of the NUMERIC field is extended to the next bit. This is repeated to further extend the LSB of the NUMERIC field until the difference in TANG vector values is outside the predetermined range. Figure 6 shows a TANG vector 36 of a first bit stream 28 and a bitmap 40 that is obtained in the IDENTIFY FIELDS step 38. The MSB of the NUMERIC field 62 is initially the bit at the 24bit position and the LSB of the NUMERIC field is initially the bit at the 21bit position. That is, the NUMERIC field 62 has a bit length of four. After the EXTEND NUMERIC field step 60, the MSB of the NUMERIC field 62 is extended to the bit at the 27bit position and the LSB of the NUMERIC field 62 is extended to the bit at the 2° bit position. The NUMERIC field 62 has beenextended to the byte boundaries and it now has a bit length of eight. In other words, the entire byte belongs to a single NUMERIC field.

[0032] The sequence 2 next proceeds to a DETERMINE BYTE ORDER step 70, wherein the device 8 determines if the bytes 72, 74 in a first bit stream 28 having at least two bytes are in a Little Endian order / format or a Big Endian order / format based on the number of fields 76 identified (F) in the corresponding bitmap 40, the sum of the TANG vector 36 values of the most significant bits (MSB) of the fields 76, and the number of loose bits that are unknown, i.e. of the VOID field type (LB). The device 8 arranges the bytes 72, 74 in both the Little Endian order and the Big Endian order. A first TANG vector 36 is obtained for the bytes 72, 74 arranged in the Big Endian order as shown in Figure 7. A second TANG vector 36 is obtained for the bytes 72, 74 arranged in the Little Endian order as shown in Figure 8. A first bitmap 40 is also obtained for the bytes arranged in the Big Endian order based on the first TANG vector 36. A second bitmap 40 is also obtained for the bytes arranged in the Little Endian order based on the second TANG vector 36. Thereafter, the device 8 obtains the following: the number of fields in the first bitmap 40, (iripLE); the sum of the first TANG vector 36 values of the MSB of the fields in the first bitmap 40, (mMSBLE); the number of VOID values in the first bitmap 40, (mLBLE); the number of fields in the second bitmap 40, (DIFBE); the sum of the second TANG vector 36 values of the MSB of the fields in the second bitmap 40, (HIMSB BE); and the number of VOID values in the second bitmap 40, (mLBBE);

[0033] The device 8 next calculates the following: a first difference given by (mFBE— mFLE); a second difference given by (mMSBBE— mMSBLE); and a third difference given by (ITILBBE- mLBLE)

[0034] The device 8 next determines if each of the first difference, the second difference and the third difference is greater than zero. If it is determined that a difference is greater than 0, the device 8 assigns a Boolean value of 1 to that difference, otherwise the device 8 assigns a Boolean value of 0 to that difference. This is expressed mathematically as follows:

[0035] The device 8 next sums up the three Boolean values and divides the result by 3 (the number of differences) to give a confidence score. If the confidence score is greater than or equal to a predetermined value, e.g. 0.5, the device 8 will determine that the bytes 72, 74 are arranged in the Little Endian order. If the confidence score is less than the predetermined value, the device 8 will determine that the bytes 72, 74 are arranged in the Big Endian order. This is expressed mathematically as follows:Confidence™ = sum of (Diffm) / length of (Diffm)„ , (Little Endian, Conf idence™ > 0.5Byte orderm=D. „ ,. ■(Big Endian, otherwise

[0036] Figure 7 shows two bytes 72, 74 of first bit streams 28 read in a Big Endian order and the TANG vector 36 and bitmap 40 associated therewith. Figure 8 shows the same two bytes 72, 74 of the first bit streams 28 read in a Little Endian order and the TANG vector 36 and bitmap 40 associated therewith. Figures 9 and 10 show the above-described calculations for determining if the two bytes 72, 74 are in a Little Endian or Big Endian format.

[0037] The sequence 2 next proceeds to a DETERMINE SIGN step 80, wherein the device 8 determines if an identified NUMERIC field is unsigned or signed, i.e. whether the value of the NUMERIC field can only be a positive value or can be both a positive or a negative value depending on a sign. The device 8 first obtains the unsigned decimal values of the occurrences of the bits of the NUMERIC field. Figure 11 shows the unsigned decimal values 80 of bits of a NUMERIC field that are known to be unsigned. And Figure 12 shows the unsigned decimal values 82 of bits of a NUMERIC field that are known to be signed. The device 8 next obtains the difference in the unsigned decimal values between adjacent pairs of the occurrences of the bits. For noccurrences of the NUMERIC field, there will be n-1 difference values 84. For the bit occurrences in Figure 11 , the set of all seven difference values 84 are of the value -1 . For the bit occurrences in Figure 12, the set of all the seven difference values are of the value -1 except one difference value which is 7. For a 3 bit NUMERIC field, the maximum value of the field is 7. The device 8 determines a NUMERIC field to be a signed NUMERIC field if this maximum value is found to exist in the set of difference values 84 as is the case shown in Figure 12. If the maximum value is not found to exist in the set of difference values, then the NUMERIC field is determined to be unsigned as in the case shown in Figure 11 .

[0038] This step 80 can be expressed mathematically as follows:The set of difference values is given by:dmaxis the maximum value of the NUMERIC field.

[0039] A NUMERIC field is determined to be signed or unsigned based on the following:

[0040] Figure 13 is a block diagram illustrating typical elements of a computing system 8 that may be appropriately programmed to function as the above-described device 8. The elements include a programmable processor 102 connected to a system memory 104 via a system bus 108. The processor 102 accesses the system memory 104 as well as other input / output (I / O) channels 110 and peripheral devices 112. The computing system 8 further includes at least one program storage device 114, such as a CD-ROM, tape, magnetic media, EPROM, EEPROM, ROM or the like. The computing system 8 stores one or more computer programs that implement the abovedescribed sequence 2. The processor 102 reads and executes the one or more computer programs to perform the sequence 2. Each of the computer programs may be implemented in any desired computer programming language (including machine, assembly, high level procedural, or object oriented programming languages). In any case, the language may be a compiled or interpreted language.

[0041] Advantageously, the above-described method is able to identify different fields in addition to NUMERIC fields. These additional fields include a FLAG field anda VARIABLE length field. The above-described method is also able to determine if an identified NUMERIC field is unsigned or signed more reliably.

[0042] Applications of the above-described method includes, but are not limited to, reverse engineering of protocol messages so as to provide clarity on data and data fields, support data sanitization / cleansing, filtering, analysis in applications such as cybersecurity, system fault and attack mitigation. And protocol reverse engineering may be extended to various other applications such, but again not limited to, simulating entity interactions, anomaly detection and network audits.

[0043] Although the present invention is described as implemented in the above described embodiment, it is not to be construed to be limited as such. It is to be appreciated that modifications and improvements may be made without departing from the scope of the present invention.

[0044] For example, although the method is described to identify fields to which bits in a payload of CAN messages belong, the method may be used to identify fields in any type of bit streams. The method may be used to identify fields represented by bits in an Internet bit stream, any time series data, any number series data, etc.

[0045] As another example, although the method is described to identify a number of field types, the method may be implemented to identify just a single field type, for example, the NUMERIC field type.

[0046] As yet a further example, although the method is described to perform an EXTEND NUMERIC FIELD step 60 to extend a NUMERIC field towards the byte boundary, the method may be implemented without such a step or extend a NUMERIC field to beyond its byte boundaries.

[0047] As yet another example, although the method is described to perform a DETERMINE BYTE ORDER step 70, the method may be implemented without performing such a step in cases where the byte order specification of the exchanged message is already known, i.e., if it is in Little Endian order / format or Big Endian order / format.

[0048] As yet a further example, although the method is described to perform a DETERMINE SIGN step 80, the method may be implemented to determine if a NUMERIC field is unsigned or signed, for example, by analysing how values of the NUMERIC field changes for unsigned or signed values, such as analysing the frequency of changes.

[0049] It should be further appreciated by the person skilled in the art that one or more of the above modifications or improvements, not being mutually exclusive, may be further combined to form yet further embodiments of the present invention.

Claims

CLAIMS1. A method of identifying at least one field in bits of a bit stream, the method comprising: obtaining a plurality of first bit streams of a same type; obtaining a number of bit flips for bits in each bit position of the plurality of first bit streams; and identifying adjacent bits of a first bit stream that do not yet belong to an identified field as belonging to a NUMERIC field if the corresponding number of bit flips are in a descending order from a least significant bit position to a most significant bit position of the adjacent bits and the adjacent bits are of at least a predetermined bit length.

2. The method according to Claim 1 , further comprising: identifying a bit of the first bit stream as belonging to a STATIC field if the corresponding number of bit flips is zero prior to identifying adjacent bits of the first bit stream as belonging to the NUMERIC field.

3. The method according to Claim 2, further comprising: identifying a bit of the first bit stream that do not yet belong to an identified field as belonging to a FLAG field if bitflips occur over one bit for that bit.

4. The method according to Claim 3, further comprising: obtaining a plurality of second bit streams of a same type as the first bit streams, the second bit stream being of a different bit length than the first bit stream; obtaining a number of bit flips for bits in each bit position of the plurality of second bit streams; identifying STATIC, FLAG and NUMERIC fields to which bits of a second bit stream belong in a same manner as that for the first bit stream based on the number of bit flips for bits in each bit position of the plurality of second bit streams; andidentifying adjacent bits of the first bit stream and the second bit stream as belonging to a VARIABLE field by comparing fields of the first bit stream and the second bit stream that are identified with a confidence level.

5. The method according to Claim 4, wherein comparing the identified fields of the first bit stream and the second bit stream comprises: matching identified fields in the first bit stream and the second bit stream to locate boundaries of the adjacent bits in the first bit stream and the second bit stream so as to identify them as belonging to the VARIABLE field.

6. The method according to Claim 5, wherein the adjacent bits in the first bit stream and the adjacent bits in the second bit stream that are identified as belonging to the VARIABLE field are of unequal bit lengths.

7. The method according to Claim 2, further comprising identifying a bit of a STATIC field to be a most significant bit of a NUMERIC field immediately adjacent to the STATIC field if the value of each of the corresponding bits in the first bit streams is zero.

8. The method according to Claim 2 or Claim 7, further comprising identifying a bit to be a least significant bit of a NUMERIC field immediately adjacent to the bit if the corresponding number of bit flips of the bit is within a predetermined range.

9. The method according to any one of Claims 2, 7 and 8, wherein the first bit stream comprises at least two bytes and identifying bits of the first bit stream as belonging to a STATIC field and a NUMERIC field are performed with the at least two bytes arranged in a little endian format and a big endian format; and wherein the method further comprises determining if the at least two bytes are arranged in one of the little endian format and the big endian format based on a number of fields identified in the first bit stream, a sum of the number of bit flips for the most significant bit of each identified field and a number of bits that do not belong to any identified field for the at least two bytes arranged in both the little endian format and the big endian format.

10. The method according to Claim 9, wherein determining if the first bit stream is of one of the little endian format and the big endian format is based on a first difference between the number of fields identified, a second difference between the sum of the number of bit flips for the most significant bit of each field and a third difference between the number of bits that do not belong to any identified field for the at least two bytes in the little endian format and the big endian format.

11. The method according to Claim 1 , further comprising determining if the bits of a NUMERIC field are of a signed or unsigned format based on a difference in value of the bits in adjacent pairs of the first bit streams in the plurality of first bit streams.

12. A device comprising a processor configured to perform the method according to any one of Claims 1 -11.

13. A program storage device readable by a computing device, tangibly embodying a program of instructions, executable by the computing device to perform the method according to any one of Claims 1 -11.

Images