METHOD FOR REDUCING THE COMPUTING TIME OF A DATA PROCESSING UNIT

Description

[0001] The invention relates to a method for reducing the computing time of a data processing device when processing data, in particular data from a data stream, wherein an input data stream consisting of data stream elements is sequentially acquired by at least one data processing device, wherein at least one elementary addressable data stream is generated from the input data stream.

[0002] Methods for processing data on data processing devices, such as computers or similar computing units for processing electronic data, are known, in which data streams composed of data stream elements are sequentially acquired by the data processing device. These data stream elements can have varying lengths. The transmitted data can often be repeated. Due to the variable lengths of the data stream elements, the data streams can only be processed if the data stream is read completely and sequentially by the data processing device. In particular, to analyze individual data stream elements, the analysis must be applied to each read data stream element. This results in a considerable expenditure of computing time and a high demand for storage space.

[0003] For example, US 2010 / 0269147 A1 describes a method for creating and using index information for video content. As another example, US 2011 / 307447A1 A1 describes the deduplication of elements from an input data stream into an output data stream.

[0004] The invention is based on the objective of proposing a method for processing data from a data stream, which reduces the computing time of a data processing device during data processing.

[0005] This problem is solved by a method having the features of claim 1. Further developments and advantageous embodiments are specified in the dependent claims.

[0006] An input data stream, such as a sensor data stream or a data stream containing log files from a web server, is sequentially captured by a data processing device, such as a processor core. The data stream can be composed of data stream elements, such as individual log file entries. These data stream elements, such as log file entries, can in turn contain elements such as a timestamp, URL, the name of a requested website or file, an HTTP response code, the transmitted file size, a referring website, descriptions of the browser used to access the website, and other entries. Due to the variable length of the data stream elements, processing individual data stream elements within the data stream is only possible if the processing is applied sequentially to the incoming data stream elements.To enable access to individual data stream elements, i.e., to establish element addressability of the data stream, at least one element-addressable data stream is generated from the sequentially read input data stream. First, each data stream element of the input data stream is divided into element parts. For example, log file entries can be divided into their various element parts as data stream elements.

[0007] The element components can now be grouped into component groups according to different categories. For example, in the case of log file entries, these component groups could include a request group, a page group, a timestamp group, a reference group, and other groups. For instance, the request group might describe the host and browser used to access a website. It can be assumed that a website is typically accessed from a specific host, i.e., a specific IP address, using the same browser. The element components grouped together can, for example, have similar, and especially identical, change frequencies, meaning they change or repeat with the same frequency within the data stream elements.Furthermore, it is possible that a share group is assigned only a single element share, and thus the share group consists of only one element share. At least one share group, or rather its value, is checked for compatibility with already recorded share groups. The value of a share group could, for example, be the browser name and IP address. From the share groups that do not match those already recorded—for example, those that contain new content compared to the already recorded share groups—an element-addressable secondary data stream is generated for the recorded input data stream. Due to the element addressability of the secondary data stream, the position of a share group within the secondary data stream can be calculated.For example, the data stream elements of the secondary data stream can have the same length, or an additional index data stream can be created for the secondary data stream. The data stream elements, i.e., the index data, of the index data stream have the same length, i.e., the same number of element segments, and each index data element contains the number of element segments output in the data stream before the assigned data stream element. The element addressability of the data stream is determined by the assignment of the index data stream. The segment group is defined by its address pointing to the secondary data stream. An output data stream is generated in which at least one element address of a segment group from the secondary data stream is contained. The sequence of data stream elements in the output data stream corresponds to the sequence of data stream elements in the input data stream.By generating an element-addressable data stream for the secondary data stream, direct access to individual data stream elements or subgroups is possible, for example, for further processing, without having to read the entire data stream sequentially. This significantly reduces the computing time of the data processing system. Furthermore, the required storage space is reduced by checking for matches between data stream elements and already recorded subgroups, since repeatedly occurring subgroups or their values ​​are stored only once.

[0008] InIn a further development of the procedure, the categories are different change frequencies and / or repetition frequencies and / or data categories of the element components. The element components can be grouped into different component groups according to various categories, for example, according to their change frequencies, i.e., their frequency of change in the data stream elements of the captured data stream. The change frequency can, for example, describe the frequency with which an element component changes in relation to the input data stream elements.

[0009] In a further development of the method, for element-addressable data streams, an element address is defined by the number of element segments output in the data stream prior to the addressed data stream element. An element address, i.e., the address of a data stream element within a data stream, is defined by the number of element segments output before the data stream element of the captured data stream. The known number and length of element segments output before the data stream element to be accessed (i.e., the addressed data stream element) in the captured data stream enables direct access to the addressed data stream element. Thus, sequential reading of the entire data stream until the data stream element to be accessed is located is unnecessary, resulting in significant time savings when accessing and processing individual data stream elements.

[0010] In a further development of the method, an index data stream is assigned to a data stream containing at least two data stream elements of different lengths, where a data stream element is considered a sequence of element parts of the same length, wherein the index data of the index data stream have the same length among themselves, wherein an index data contains the number of element parts output in the data stream before the assigned data stream element, wherein the order of the index data corresponds to the order of the assigned data stream elements in the data stream, and wherein the element addressing of the data stream is given via the assignment of the index data stream.Element addressability of a data stream can be achieved if all data stream elements have the same length, meaning the data stream elements have the same number of element segments, with each segment having the same length. This allows the position of a data stream element or group of segments within the data stream to be calculated by multiplying the length of the data stream elements accordingly. For example, the position of the nth data stream element can be calculated by multiplying the length of a data stream element by n. Often, data streams contain data stream elements of different lengths, i.e., different numbers of element segments. In this case, a data stream element can be considered a sequence of element segments of the same length.To ensure addressability of a data stream with elements of varying lengths, an index data stream is assigned to the data stream. An index data stream consists of a sequence of index data, where the index data have the same known length. Each data stream element, or each group of elements composed of element segments, is assigned an index datum. The index datum assigned to a data stream element contains the number of element segments that precede the data stream element in the data stream to which the index datum is assigned. The order of the index datums corresponds to the order of the assigned data stream elements within the data stream.The element addressability of the data stream is ensured through the assignment of the index data stream, since the position of an index datum assigned to a data stream segment to be called within the index data stream can be calculated because the index datums have the same length. The index datum contains the number of element segments that precede the data stream element to be called, thus revealing the position of the data stream element to be called within the data stream. In particular, an index data stream can be assigned to a sub-data stream of an input data stream. The sub-data stream contains only data stream elements of the input stream that have not yet been recorded. Each data stream element of the sub-data stream is assigned an index datum. The index datum contains the number of element segments that precede the data stream element to be called within the sub-data stream.Because the index data stream contains index data of the same length, the position of each index data point can be calculated. In an output data stream, the addresses of the share groups in the sub-data stream can be included by outputting the position of the respective index data points assigned to the share groups within the index data stream. Thus, the index data point assigned to a share group can be accessed, with the element address of the share group in the sub-data stream specified in the accessed index data point.

[0011] In a further development of the method, the maximum available number of data stream elements of a data stream is defined by a data interval of the data stream, and addressable data stream elements of a data stream located within a data interval refer to data stream elements of the addressed data stream located within a data interval. The maximum number of data stream elements available at any given time is defined by a data interval of the data stream. A data interval thus forms a data window. A data interval can, for example, be defined by a ring buffer. Data stream elements lying outside the data window, or groups of elements formed from the element components of the data stream elements, are recognized as new during the recognition check, i.e., when checking whether a group of elements, or the value of the group of elements, has already been recorded or not.Addressable data stream elements arranged within a data window in a data stream refer to data stream elements of the addressed data stream arranged within a data interval. The referencing data stream elements are thus spatially close, for example within a storage device, enabling a fast reference from an addressing data stream element to the addressed data stream element. In particular, the referencing data stream elements can be stored in a cache.

[0012] In a further development of the method, a secondary data stream with the same number and sequence of data stream elements is generated for an existing data stream. This secondary data stream contains modified data stream elements that are dependent on the data stream elements of the original data stream. The data stream elements of the secondary data stream are addressable via the same element addresses as the corresponding data stream elements of the original data stream. For example, a secondary data stream can be generated that contains only previously uncaptured subgroups of a data stream. These subgroups or element subgroups can be further subdivided or combined into new groups. A further data stream can be generated for a secondary data stream, and this further data stream contains the same number and sequence of data stream elements.The data stream elements of the secondary data stream depend on the data stream elements of the secondary data stream, in that the data stream elements of the secondary data stream are derived from the data stream elements of the secondary data stream, for example, through further decomposition or regrouping. The data stream elements of the secondary data stream can be addressed via the same element addresses as the corresponding data stream elements of the original data stream, since the number and order of the data stream elements are preserved.

[0013] In a further development of the method, by preserving the sequence of data stream elements of the input data stream in the output stream, the data stream addressing the original input data stream also refers to the corresponding data stream elements of the output data stream, even when the method is applied recursively. The sequence of data stream elements of the input data stream is preserved in the output data stream. Thus, the data stream addressing the original input data stream also refers to the corresponding data stream elements of the output data stream, even when the method is applied repeatedly, i.e., recursively nested. Repeated application of the method to the secondary data streams is therefore possible. In particular, recursive execution of the method can also be carried out at a later time or using a further data processing device.

[0014] In a further development of the process, the data streams are processed in independent blocks, and this processing is carried out separately both spatially and temporally. The further processing, for example of secondary data streams, in particular a regrouping into further component groups or the further decomposition of element components, can be carried out separately both temporally and spatially, i.e., for example, at a later time than the first execution of the process and using a separate data processing device.

[0015] In a further development of the method, at least one mapping, in particular a hash value, is assigned to a fraction group for the conformance check. This hash value is a numerical hash value calculated from the fraction group. Identical hash values ​​are calculated for identical fraction groups. A contiguous memory area is maintained as a sequence of memory locations for storing element addresses, where each memory location can contain an element address or be marked as unused. A memory location is assigned to a fraction group via the hash value. If a memory location is marked as unused, the fraction group is considered unrecognized and output in the assigned sidestream. The address of the fraction group is stored in the sidestream of the memory location, and this address is subsequently used as the result of the fraction group check. One possibility for a conformance check of fraction groups or...The process of performing the mapping of values ​​of fractional groups, with previously recorded fractional groups or their values, involves assigning a mapping, in particular a numerical hash value, to a fractional group or to a value within a fractional group. Identical hash values ​​are calculated for identical fractional groups, i.e., fractional groups with identical values. In particular, a unique mapping of the value of a fractional group to a hash value is possible. Based on the hash value, and especially depending on the hash value, a memory location in a memory area is assigned to a fractional group or its value, specifically the element address of a fractional group. The memory locations in the memory area are marked either as occupied by an element address or as unoccupied, i.e., free. If a fractional group has a free memory location assigned to it, the fractional group or its value is considered new and therefore not yet recorded.viewed undetected and output to the assigned secondary data stream. In The free memory space is used to store the element address of the share group in the secondary data stream, i.e., the element address addressing the secondary data stream. This stored address is used to check for matches between other share groups, or between the values ​​of the share groups, by referencing the share group in the secondary data stream.

[0016] In a further development of the procedure, when a memory location is marked as occupied, the subgroup addressed by the element address specified in the memory location is checked for a match with the subgroup to be stored, and if there is a match, the address specified in the memory location is output to the output stream.

[0017] In a further development of the invention, if the fraction group does not match the fraction group addressed by the occupied memory location, an alternative memory location is assigned to the fraction group to be stored by means of the hash value. A further compatibility check is performed for this alternative memory location, and if further mismatches occur, further alternative memory locations are assigned. If the fraction group, or the value of the fraction group, assigned to the element address in the memory location does not match the recorded fraction group, or the value of the recorded fraction group, another memory location is assigned to the fraction group to be stored via the hash value. A compatibility check is again performed for this additional memory location. If the fraction group to be stored, or its values, match the fraction group addressed by the address stored in the additional memory location, the fraction group to be stored is not assigned.The value of this hash value determines which element address is stored in the memory location and output to the data stream. If there is a mismatch, another memory location can be selected using the hash value. The number of cycles, i.e., the number of alternatively allocated memory locations, can be specified in the settings. For example, if the subgroup to be stored does not match the subgroup addressed in the third allocated memory location, the element address stored in the third memory location can be replaced by the element address of the subgroup to be stored.

[0018] In a further development of the invention, if the component group does not match the component group addressed by the occupied memory location, the address of the component group is stored in the memory location via a sidestream, replacing the original content of the memory location. If the component group to be stored does not match the component group addressed by the element address in the allocated memory location, the element address stored in the allocated memory location can be replaced by the element address of the component group to be stored. This replaces element addresses referring to older data with those referring to data previously output by the data stream, since it can be assumed that this data is no longer needed for the current processing.

[0019] In a further development of the invention, when memory locations are repeatedly occupied, the memory location with the smallest element address is defined as unoccupied, and if the memory location contains an address that lies outside the available sidestream interval, the memory location is defined as unoccupied. For example, if the memory locations assigned to the share group are repeatedly occupied during multiple executions of the method—that is, if the same memory locations are assigned to a share group cyclically in succession—the memory location with the smallest element address can be defined as unoccupied. The smallest element address of an addressed share group indicates that the share group was output in the data stream before the share groups addressed in the subsequent memory locations, and thus the data of the share group is older.Thus, the element address of the older share group can be overwritten with the element address of the share group to be saved.

[0020] For the conformance check, a multi-stage conformance check is applied, whereby the value of a data stream element is first processed in a global area assigned to at least one group of share groups and converted as a result into an address in the side stream. In a second stage, this address is assigned to at least one subgroup from the group of share groups assigned to the area and side stream. This side stream then contains these addresses instead of directly share group values ​​and is thus considered an index in the main side stream, and this additional side stream therefore only contains addresses that have occurred in this subgroup.

[0021] In a further development of the procedure, a multi-stage conformance check is used for conformance verification. In the first stage, the value of a data stream element is captured. The value of the data stream element consists of element components. An address in a first sub-data stream is assigned to the value of the data stream element, which consists of element components. In the second stage, at least one group of the addresses of the element components in another sub-data stream is assigned to this address. This second sub-data stream contains the addresses of the element components and thus functions as an index to the first data stream. For example, data stream elements could be server log files containing, for instance, a timestamp, the name of a client used, the browser used, the website accessed, and a timestamp.The elements of a data stream are determined by the client, browser, and website used. For example, a timestamp can be separated from the other elements and processed separately. There are various possible combinations of client, browser, and website. Each combination is assigned an address in a substream. A data stream element, in this case a log file, can be assigned an address in a substream, allowing the combination of elements contained in the data stream to be addressed via an index.

[0022] To create and assign a check level, a value or an address within a group is used instead of a group type; in this level, values ​​of a second share group are checked for matches, and only element addresses corresponding to a combination of the level value and the share group value are output in the assigned side stream.

[0023] In a further development of the procedure, a data stream containing data stream elements is recorded, wherein the data stream elements are composed of different element components, the element components of all data stream elements can be sorted into categories, the element components of other categories that occur together with an element component of a first category in a data stream element are checked for similarity, and the element components that occur together with the element components of the first category in other data stream elements are output in a secondary data stream.

[0024] For example, a data stream can be composed of data stream elements, where the data stream elements could be, for instance, a server's log files. These data stream elements, such as the log files, can consist of element components from various categories. For example, the name of the client used could be the first category, and the browser name or the accessed website could each be a further category. Each client value, such as Client A or Client B, can then be assigned an element-addressable substream containing, for example, the browsers used by each client type or the websites accessed by each client type. This enables very fast access to information such as which websites were accessed by a particular client type or which browsers were used by a particular client type.

[0025] Each distinct value from one subgroup is assigned the current value or address of a second subgroup, and this is stored in a memory area sorted according to the value or address of the first group. This memory area is considered an element of a new input data stream for a further stage of the procedure, in which a conformity check is applied again according to the division into groups, and as an output stream result, patterns and sequences of values ​​in groups are recognized.

[0026] In a further development of the procedure, a data stream element consists of element parts, the element parts of a data stream element can be sorted into at least two categories, each element part of a first category is assigned at least one element part of a second category, the assignment of the element parts of a data stream element is checked for agreement with the assignment of the element parts of the other data stream elements, and an address in a secondary data stream is assigned to the assignment of the element parts.

[0027] The elements of a data stream are sorted into two categories. For example, the elements in the first category might be the names of various electrical devices, while the elements in the second category consist of the states of the devices, such as whether they are switched on or off. The current state of all devices—that is, the mapping of the elements in the first category to the elements in the second category—can be assigned to an address in a secondary data stream. This allows recurring states, such as when devices A and B are switched on and devices C and D are switched off, to be recognized efficiently and transmitted with minimal data transfer.

[0028] In a further development of the procedure, procedural steps are applied within a stage or in cascading stages, and thus separately in time and space.

[0029] The inventive method is further explained below using an exemplary embodiment: Example: Apache web server access log file

[0030] For your information: The log file viewed here is completely randomly generated.

[0031] The following symbols are used below: xyz - Identifier (e.g., sub-element name, group, ...) ... - Synonym for arbitrary content |xyz| - Sub-element @xyz@ - Group / Subgroup [...] - Denotes an array or grouped data {...} - Denotes a recognition level / map ~...~ - Denotes a stream xyz[...] - Form for named structure, here: an array xyz with elements ... xyz[a,b,c] - Specialized form, here: array xyz with elements a, b, and c xyz{} - Recognition level / Map ~xyz~ - Stream / Sub-stream / Side stream / Output stream abc>~xyz~ - Address of an element abc in the stream xyz

[0032] The format of these log files is usually as follows: Example:

[0033] 192.168.178.13 - - [02 / Jun / 2015:22:49:39 +0100] "img / nord.png" 200 1928384 "www.blog_ueber_dinge.de / wir / im / urlaub / 2014 / April / links" "Mozilla / 5.0 (Linux; Android 4.4.2; A3-A20 Build / KOT49H) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 57.0.2987.132 Safari / 537.36" "-" 192.168.178.13 - - [02 / Jun / 2015:22:52:35 +0100] "img / whg1.png" 200 283212 "img / nord.png" "Mozilla / 5.0 (Linux; Android 4.4.2; A3-A20 Build / KOT49H) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 57.0.2987.132 Safari / 537.36" "-" 192.168.178.192 - - [02 / Jun / 2015:23:01:06 +0100] "img / whg2.png" 200 483722 "www.reisebuero.net?kunde=12382ajablkdsu12" "Mozilla / 5.0 (Linux; Android 4.4.2; de-at; SAMSUNG GT-I9301I Build / KOT49H) AppleWebKit / 537.36 (KHTML, like Gecko) Version / 1.5 Chrome / 28.0.1500.94 Mobile Safari / 537.36" "-" 192.168.178.192 - - [02 / Jun / 2015:23:06:52 +0100] "photos / 0001.png" 200 -1 "img / whg2.png" "Mozilla / 5.0 (Linux; Android 4.4.2; de-at; SAMSUNG GT-I9301I Build / KOT49H) AppleWebKit / 537.36 (KHTML, like Gecko) Version / 1.5 Chrome / 28.0.1500.94 Mobile Safari / 537.36" "-" Abstracted, each line (separated by "\n") contains the following parts / sub-elements: |host| - - [|timestamp|] "|request|" |code| |size| "|referer|" "|agent|" "|aux|" . Definition:

[0034] |host| : the URL / address of the requesting party |timestamp|: timestamp (format: day / month / year:hour:minute:second offset from Greenwich Mean Time) Request Result code File size Browser information |request| : the requested web page / file |code| : the HTTP response code, e.g., 200 for OK |size| : submitted file size |referer| : linking page previously viewed by the visitor |agent| : description of the browser / tool used to visit the web page |aux| : ?

[0035] From this observation and logical analysis, the following division / grouping was determined: Group @req@: |host| + |agent| (it can be assumed that requests from $host are largely made using the same browser $agent => similar change frequency) Group @pag@: |request| + |code| + |size| (assumption: identical file requests are usually answered with the same response code $code and the same file size $size) Group @utc@: |timestamp| (actually not a group; timestamps are not recognized and are handled separately, repetition is not expected) Group @ref@: the linking page. Group @aux@: |aux| is ignored in this consideration (in the present example, this field is always empty).

[0036] Based on this, appropriate data structures are now created / storage areas are provided: {RAW}: First recognition stage, here all relevant groups (@req@, @pag@, @ref@) are processed. In this stage, the actual recognition of the group values ​​takes place; values ​​are converted into addresses in the sidestream ~RAW~. A data structure of 1000 memory locations (slots) is created for addresses in the sidestream. Additionally, a 1MB ring buffer is maintained for the value sidestream (containing the last 1MB of values ​​for the match check). Furthermore, a 1000-element ring buffer is maintained for the index sidestream (containing the 1000 offsets to the value sidestream for the match check; the values ​​are of variable length; the index sidestream serves to establish element addressability). {Req}: Subsequent recognition stage for creating a list of visitors. Here, the previously determined addresses are processed again.and the address assigned to this group in the (address) sidestream ~Req~ is converted into a data structure, creating an area of ​​100 memory locations (slots) for two addresses each. {Pag}: downstream recognition stage for creating a list of visited pages. Here, the previously determined addresses are processed again, and the address assigned to this group in the (address) sidestream ~Pag~ is converted into a data structure, creating an area of ​​100 memory locations (slots) for two addresses each. {Ref}: downstream recognition stage for creating a list of visited pages. Here, the previously determined addresses are processed again, and the address assigned to this group in the (address) sidestream ~Ref~ is converted into a data structure, creating an area of ​​100 memory locations (slots) for two addresses each.

[0037] The generated data structures are pre-populated with a value (hex 0xFFFFFFFF) to indicate an "empty" slot.

[0038] Since values ​​in the "group" @utc@ are expected to change constantly, they are not subjected to any recognition; instead, they are converted into a Unix timestamp, which can be stored as a 32-bit unsigned integer (4 bytes) and is specified as seconds since January 1, 1970.

[0039] Furthermore, from a data reduction perspective, the ability to convert to a 4-byte value is equivalent to recognizing and converting to an address (also 32 bits).

[0040] All determined group addresses are then combined with the converted timestamp to form an output element and appended to the output stream.

[0041] The resulting structure after processing is as follows: RAW~firstReq,firstPag,firstRef,secondReq,secondPag...~ Req~firstReq>RAW,secondReq>RAW...~ Pag~firstPag>RAW,secondReq>RAW...~ Ref~firstRef>RAW,secondRef>RAW...~ Output-[timestamp, first>Req, first>Pag, first>Ref],[timestamp, second>Req, second>Pag, second>Ref],...~

[0042] After defining / generating the necessary data structures, the log file is now processed element by element as an input stream. Entrance element:

[0043] 192.168.178.13 - - [02 / Jun / 2015:22:49:39 +0100] "img / nord.png" 200 1928384 "www.blog_ueber_dinge.de / wir / im / urlaub / 2014 / April / links" "Mozilla / 5.0 (Linux; Android 4.4.2; A3-A20 Build / KOT49H) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 57.0.2987.132 Safari / 537.36" "-" Division into sub-elements:

[0044] |host| : 192.168.178.13 |utc| : 02 / Jun / 2015:22:49:39 +0100 |request|: img / nord.png |code| : 200 |size| : 1928384 |referer|: www.blog_ueber_dinge.de / wir / im / urlaub / 2014 / April / links |agent| : Mozilla / 5.0 (Linux; Android 4.4.2; A3-A20 Build / KOT49H) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 57.0.2987.132 Safari / 537.36 |aux| :- Grouping:

[0045] @utc@ : 02 / Jun / 2015:22:49:39 +0100 / / |utc|; corresponds to 1433281807 (seconds since 1.1.1970) @req@ : [192.168.178.13,Mozilla / 5.0 (Linux; Android 4.4.2; A3-A20 Build / KOT49H) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 57.0.2987.132 Safari / 537.36] / / |host|,|agent| @pag@ : [img / nord.png,200,1928384] / / |request|,|code|,|size| @ref@ : www.blog_ueber_dinge.de / wir / im / urlaub / 2014 / April / links / / |referer| First-stage detection (stage {RAW}):

[0046] 1) Generating a hash value for @req@: h = hash("[192.168.178.13,Mozilla / 5.0 (Linux; Android ....]") = 2003304065 2) Selecting a slot via h: h modulo 1000 = 65 = slotA / / we have chosen an array with 1000 slots as the data structure 3) Checking the slot: if marked as "empty" (0xFFFFFFFF) or address outside the index ring buffer area or index value (offset) outside the value ring buffer area: Output the value of @req@ to the ~RAW~ value stream by appending it to the value ring buffer. Output the (byte) address of the value in the value stream (ring buffer) to the index stream (index ring buffer). Store the address (=number of the element in the index stream) in the determined slot and also Return as result otherwise (valid address available),Determine the associated value and compare it with the current value of @req@. If the values ​​match: return the slot content (=address). If the values ​​do not match: discard the slot content. Output the value of @req@ to the ~RAW~ value stream by appending it to the value ring buffer. Output the (byte) address of the value in the value stream (ring buffer) to the index stream (index ring buffer). Store the address (=number of the element in the index stream) in the determined slot and also return it as a result. Since the first value is considered here, the result is address 0. Recognition in the second stage (here {Req}):

[0047] 1) Selecting a slot using the previously determined address req>RAW (=0): req>RAW modulo 100 = 0 = slot / / we have chosen a structure with 100 elements here. 2) Checking the slot (position 1 of the slot, 2 positions per slot): if empty (0xFFFFFFF): Output req>RAW in the ~Req~ sidestream. Store req>RAW in the first position of the slot (2 positions per slot). Store the address (=number in the sidestream) in the second position of the slot and return it as a result. If equal: Output the address in the second position of the slot as a result. If not equal: Output req>RAW in the ~Req~ sidestream. Store req>RAW in the first position of the slot (2 positions per slot). Store the address (=number in the sidestream) in the second position of the slot and return it as a result.

[0048] The same procedure is followed for the other groups @pag@ and @ref@.

[0049] The result is three addresses and a timestamp, each as an unsigned integer 32bit (4 bytes).

[0050] These are now appended to the output stream as a group [timestamp, first>Req, first>Pag, first>Ref].

[0051] The process then starts again from the beginning with the next element of the input stream.

[0052] The invention will now be explained in more detail with reference to an embodiment shown in the drawing. The figures show: Figure 1: a schematic representation of the assignment of an index data stream to a data stream; Figure 2: a schematic representation of the matching of individual share groups by assigning a hash value; Figure 3: a schematic representation of the separate processing of share groups; Figure 4: a schematic representation of the exchange of secondary data streams with other data streams.

[0053] In Figure 1This schematically illustrates, using an exemplary embodiment, how an index data stream 3 can be assigned to a data stream 1, which has at least two data stream elements 2 of variable length. Generally, a single piece of data, i.e., a data stream element 2 of a data stream, has a variable length. This could be, for example, the file paths or, later on, the file content within data streams, in this case, a list of file paths. In this example, the data stream elements are words that can be of different lengths, i.e., they can contain different numbers of letters. For targeted access, the entire data stream 1 must be read sequentially to determine the start, length, and end of the data stream. To enable index-based access, an additional, element-addressable index data stream 3 is created during data acquisition, for example, when reading the file.Index data stream 3 can contain offsets of the individual data points from the actual data stream 1. This new index data stream 3 consists entirely of entries, i.e., index data 4 of the same length, for example, 32-bit integers. The position of the nth entry can thus be determined by a simple calculation, in this case n * 4 bytes, which then references the beginning of the associated data point in the data stream. This index 4 in the new data stream 1 can now be understood as an additional identifier, i.e., an ID of the data point, although so far no recognition of identical data segments is possible.

[0054] The actual data stream 1 can now be processed further. For example, when index data stream 3 is replaced by a correspondingly adapted, further processed data stream, the same index 4 from index data stream 3 always references the same, but now further processed, data record—for example, the file content instead of the file path. As a basic structure and concept for referencing, the resulting ID, due to its sequential nature (e.g., it can be a continuous number), can also be used directly for synchronization or lazy loading / processing concepts. This means that when a specific index / ID is queried, paging structures (similar to virtual memory concepts) can be used to check whether (further processed) data is already available, and thus only load / process it when needed.

[0055] In Figure 2The matching of individual element groups 5, in this example individual words, is schematically represented by assigning a hash value 6. The following section explains the background to recognizing identical data or file contents as effectively as possible. For this purpose, the analysis separates the highly variable element components, such as the file path, timestamp, or similar information, from the recurring element components 5, such as the actual file contents or, in this example, words, with the initial focus being on the recurring element components 5.

[0056] The method is based on an arrangement, a so-called array, of the index entries defined above. This arrangement is called a MAP. This MAP is connected to the combination of data streams and indexed side streams defined above.

[0057] For each incoming data record, or as described above for the previously extracted element portions 5, here for each word, a hash value 6 is calculated using common methods such as xxhash. This value is then used to select a storage location 7, a so-called slot in the map. For example, the slot number can be calculated as: slotNr = hash modulo map size. If this slot is still unoccupied, or if the index or the referenced substream is outside a predefined data interval, the data is appended as a new entry in the OFF / DAT streams, i.e., the index data stream or substreams. The corresponding index, i.e., the element address, is stored in the assigned slot, and this index is then appended to the map output stream. The data interval can be a sliding data window of available entries in the OFF / DAT stream, measured from the last current entry, e.g.,1000 entries in the cache or, for example, 1GB of data in the cache.

[0058] If the slot is not empty, the content 9 referenced via the contained index 8 is first compared with the current content. If there is a discrepancy 10, a new slot 11 is selected from the hash value 6, for example, by recombination of the used bytes or other division, and the process is then continued with this new slot 11. This is repeated until a free or old slot is found or further repetition is deemed unnecessary. A trade-off can be made between the risk of non-recognition and the time required for repetition. For example, if the fraction group 5 is repeatedly assigned to occupied slots 7 and 11, an occupied slot 11 can be overwritten with the fraction group 5 to be stored. In particular, the fraction group 5 to be stored can be compared with the content 13 referenced via the index 12 contained in the reassigned slot 11.If there is no match (14), slot 11 can be overwritten with the portion group to be stored (15). If there is a match, the index is returned in the MAP output stream. The size of the cache window, the MAP, the number of iterations, and the quality of the hashing process are configurable settings for optimal detection of identical data. This makes it possible to apply the method to any data set with limited resources, such as limited storage space, and, if necessary, to achieve near 100% detection through multiple iterations. For example, "collisions" via the hash value are considered relatively rare, depending on the size of the MAP.

[0059] In Figure 3The schematic representation of the separate processing of component groups is shown. As already described, potentially repeating element components 16, i.e., as shown here, individual words such as file contents, are separated in advance from potentially unique, non-repeating element components 17, such as file paths, here timestamps, or similar. This separation takes place in a higher-level processing step. The individual parts are then, if necessary, assigned to their own processing streams according to Fig. 2 supplied or directly reused and added in result (stream) 25, referencing the other partial results.

[0060] In Figure 4This is a schematic representation of the exchange of secondary data streams with other data streams. For example, after an initial processing run, the data can be further processed. By exchanging the OFF data streams, the data entries referenced by indexes or element addresses can each point to additional or modified data. It would be conceivable, for example, to annotate the files with keywords such as resolution, subject, and locations (e.g., for photos) through further analysis, such as image recognition. An analysis is then only necessary once, as the results can be reused directly if the process is repeated. In the example, file paths to identical content could be marked by links to the first recognized file; the actual content only needs to be stored once.For example, repeating element components 16, such as file content or, in this example, sentences consisting of words, can be separated from non-repeating element components 17, such as timestamps. An index data stream 18 can be created for the data stream 16 containing the repeating element components, which references the sub-data stream 19. An output data stream 20 can be generated, which references the sub-data stream 19. The sub-data stream 19 can be further divided into repeating element components, here individual words occurring in the sentences, which can be grouped into another sub-data stream 21. Another index data stream 22 can reference the sub-data stream 21. For this purpose, another output data stream 23 can be generated, which references the sub-data stream 21.Thus, output data stream 23, sub-data stream 21, and index data stream 22 replace sub-data stream 19 with index data stream 18. Likewise, the element components can be sorted into categories 24, thereby extending sub-data stream 21.

Claims

1. Method for reducing the computing time of a data processing device during the processing of data of a data stream, consisting of the steps, that an input data stream (1) consisting of data stream elements (2) is detected sequentially by at least one data processing unit, that at least one element-addressable data stream (3) is generated from the input data stream (1) in order to enable access to individual data stream elements, that each data stream element (2) of the input data stream (1) is divided up into element components, that the element components are grouped into component groups according to different categories, that element components of at least one component group are checked for coincidence with already detected element components of the corresponding component group, wherein an element-addressable secondary data stream (3) is generated from the element components of the component group not coinciding with already detected element components of the at least one component group, which enables the calculation of the position of an element component of the component group in the secondary data stream, wherein the element component of the component group is subsequently defined via its address referring to the secondary data stream (3), that an output data stream is generated, wherein the output data stream includes the addresses of the element components of the at least one component group in the secondary data stream (3), wherein the sequence of the data stream elements of the output data stream corresponds to the sequence of the data stream elements (2) of the input data stream (1), and wherein element components of the component group that are identified as coinciding with the already detected element components of the at least one component group are not stored in the secondary data stream again.

2. Method according to Claim 1, characterized in that the categories are different change frequencies and / or repetition frequencies and / or data categories of the element components.

3. Method according to either of Claims 1 or 2, characterized in that, in the case of element-addressable data streams (3), an element address is defined by the number of data stream elements output in the data stream prior to the addressed data stream element.

4. Method according to Claims 1 to 3, characterized in that a data stream having at least two data stream elements of different lengths, wherein a data stream element is regarded as a sequence of element components of equal length, is assigned an index data stream, wherein the index data stream (3) has index data (4) and the index data (4) of the index data stream (3) being of the same length among each other, wherein an index data (4) contains the number of element components of the data stream components output in the data stream prior to the assigned data stream element, wherein the sequence of the index data (4) corresponds to the sequence of the assigned data stream elements (2) in the data stream (1) and wherein the element addressing of the data stream (1) is given by the assignment of the index data stream.

5. Method according to any one of Claims 1 to 4, characterized in that the maximum available number of data stream elements (2) of a data stream (1) is specified by a data interval of the data stream (1), that addressable data stream elements of a data stream that are arranged within a data interval refer to data stream elements of the addressed data stream arranged within a data interval.

6. Method according to any one of Claims 1 to 5, characterized in that a further data stream with the same number and sequence of data stream elements is generated for a data stream, wherein the further data stream comprises data stream elements that are modified with respect to the original data stream and dependent on the data stream elements of the original data stream, wherein the data stream elements of the further data stream are addressable via the same element addresses as the corresponding data stream elements of the original data stream.

7. Method according to any one of Claims 1 to 6, characterized in that, by retaining the sequence of the data stream elements of the input data stream in the output stream, the data stream addressing the original input data stream also refers to the associated data stream elements of the output data stream during recursive application of the method.

8. Method according to any one of Claims 1 to 7, characterized in that the data streams are further processed in mutually independent blocks and in that the further processing takes place in both a spatially and temporally separated manner.

9. Method according to Claims 1 to 8, characterized in that in order to carry out a coincidence check, at least one component group (5) is assigned at least one mapping, in particular a hash value (6), that this hash value (6) is a numerical hash value (6) calculated from the component group (5), that equal hash values (6) are calculated for identical component groups (5), that a contiguous memory range is reserved as a sequence of memory locations for storing element addresses, wherein each memory location can contain an element address or is marked as unoccupied, that a component group (5) is allocated a memory location (7) via the hash value (6), that in the case of a memory location marked as unoccupied, the component group (5) is regarded as unidentified and is output in the associated secondary data stream, that the address of the component group in the secondary data stream is stored in the memory location, that this address is subsequently used as a result of the check of the component group.

10. Method according to Claim 9, characterized in that, in the case of a memory location (7) marked as occupied, the component group (9) from the secondary data stream addressed by the address (8) specified in the memory location is checked for coincidence with the component group (5) to be stored and that in case of coincidence, the address specified in the memory location is output to the output stream.

11. Method according to either of Claims 9 or 10, characterized in that, if in case of non-coincidence (10) between the component group (5) to be checked and the component group (9) addressed by the occupied memory location (7), the component group (5) to be stored is allocated an alternative memory location (11) by means of the hash value (6), that a further coincidence check (14) is carried out for this alternative memory location (11) and that additional alternative memory locations are allocated in case of further non-coincidences.

12. Method according to either of claims 9 or 10, characterized in that, in case of non-coincidence (14) between the component group (5) and the component group (13) addressed by the occupied memory location (11), the address of the component group (5) in the secondary data stream is stored in the memory location, wherein the original content of the memory location (11) is replaced (15).

13. Method according to any one of Claims 9 to 12, characterized in that, in the case of recurring allocation of the memory locations (7, 11), the memory location with the lowest address is defined as unoccupied and that, in the case of an address contained in the memory location and located outside of the available secondary data stream interval, the memory location is defined as unoccupied.

14. Method according to any one of Claims 1 to 13, characterized in that a multi-stage coincidence check is applied for coincidence checking, wherein in a first stage the value of a data stream element is detected, wherein the value of the data stream element consists of element components, that the value of the data stream element consisting of element components is allocated an address in a first secondary data stream and that in a second stage at least one group of the addresses of the element components in a further secondary data stream is assigned to this address, wherein this further secondary data stream contains addresses of the element components and thus acts as an index into the first data stream.

15. Method according to any one of Claims 1 to 14, characterized in that a data stream comprising data stream elements is detected, wherein the data stream elements are composed from element components different from each other, that the element components of all data stream elements can be sorted into categories, that the element components of a first category occurring together with an element component of further categories in a data stream element are checked for coincidence and that the element components occurring together with the element components of the first category in further data stream elements are output in a secondary data stream.

16. Method according to any one of Claims 1 to 15, characterized in that a data stream element consists of element components, that the element components of a data stream element can be sorted into at least two categories, that each element component of a first category is assigned at least one element component of a second category, that the assignment of the element components of a data stream element is checked for a match with the assignment of the element components of the further data stream elements and that an address in a secondary data stream is allocated to the assignment of the element components.

17. Method according to any one of Claims 1 to 16, characterized in that method steps are applied within one stage or in cascading stages, and are thus temporally and spatially separated.