METHOD AND INTERMEDIATE STORAGE DEVICE FOR VEHICLE MEASUREMENT DATA

Description

[0001] A trend is emerging towards the automated, real-time generation of ever-larger data sets from sensors, which are then analyzed as quickly, comprehensively, and globally as possible. This trend is expected to continue, and perhaps even intensify disproportionately, particularly in the context of automated driving, but not exclusively.

[0002] The complexity of electronic control units (ECUs) and local data transmission buses has increased more sharply in recent years than the bandwidth of global communication networks due to the digitalization and automation of motor vehicles, aircraft, ships, and production facilities. Many of the message details and measurement information generated locally in these machines, also known as measurement signals or measurement data, can therefore only be partially, periodically aggregated, and / or stored in local memory, despite being extremely relevant for safety and economic reasons. In this context, signals are stored, for example, in the diagnostic memory of individual ECUs, a black box, or dedicated data loggers.They are therefore initially ineligible for "real-time" processing by advanced cloud-based intelligence (cloud = interconnected data centers used by multiple tenants and therefore operated very economically). It is expected that the expansion of available bandwidth will not be able to keep pace with the increasing volume of measurement data for the foreseeable future, meaning that innovative applications / analyses face enormous cost barriers.

[0003] In order to better describe the present technical field, mathematical definitions are introduced at appropriate points in the text and then subsequently used to present the facts.

[0004] Regarding the recording of digital signals, it is defined as follows: Let ( T = { t 1 , t 2 , ...}, <) a strict order of discrete points in time, C = { C 1 , C 2 ,...} a number of channels as well as ( M = { m 1 , m 2 , ...}, +̇, 0̇) a monoid of messages (when interpreted numerically: values). A signal trace S ⊆ C × T × M represents a set of signals such that ∀ ( c, t, m i ) , ( c, t, m j ) ∈ S ⇒ m i =m j . A signal trace therefore represents a partial function on channels and times.

[0005] The preferred method for obtaining signal traces from complex systems with a wide variety of control units and bus standards is based on the use of dedicated multi-channel control units, which are referred to as loggers.

[0006] The loggers route all signals present on a message bus - a so-called sample. σ ( t ) = {( c, t, m ) ∈ S} - to a memory. This occurs at predetermined intervals, which are based on the clock frequency of the respective communication channel or internal states (e.g., per motor revolution). The data is permanently stored in the memory and can be read out at a later time. The logger can either encompass the memory unit or be in a data connection with the memory unit.

[0007] The design of such a logger is determined by the real-time requirements of the respective technical field. Other factors can include disaster recovery and cost considerations. Typically, the samples first traverse a small main memory area (time-limited ring buffer) where, depending on the signal type, they are pre-transformed (e.g., logarithmically projected) or pre-compressed (e.g., multi-channel transposed) before finally being sequentially written to a shock-resistant long-term storage medium (EEPROM, magnetic tape, or solid-state drive - SSD). The process of extracting low-frequency data from a high-frequency data stream in the form of contiguous data blocks is called flushing.

[0008] The information recorded by a logger is qualitatively and quantitatively superior to conventional telematics monitoring methods. In telematics monitoring, predefined events (abstract signals derived from the analysis / observation of individual source signals) are exchanged with a control center via current mobile communication standards using pre-programmed gateway control units. However, telematics solutions for optimized event processing are only partially relevant for the raw signals underlying this invention, because the often dynamically defined filtering, aggregation, and further analyses, which arise particularly in automated driving, engineering, and safety contexts, cannot be pre-programmed in this way.

[0009] The data recorded by the logger is subsequently also referred to as signal or signal track data. These terms reflect the fact that different sensors and bus systems transmit their respective measurement data to the logger in their own tracks or formats. For example, engine control units (ECUs) on the CAN bus generally generate data with a different format than video or environmental sensors on the FlexRay bus. These signal tracks / formats therefore generally exhibit different encoding and decoding characteristics.

[0010] Depending on the industrial environment and influenced by specific concepts of plant, component, and logger manufacturers, a variety of formats for recording and transmitting digital signals and measurements have consequently emerged. In the automotive sector, these include the "Measurement Data Format" and the "EB Assist Automotive Data and Time-Triggered Framework." In the area of ​​internet communication (Ethernet bus), which is now also used in modern vehicles, the "packet capture" structure plays a dominant role, and a variant for the automotive industry also exists.

[0011] These formats share the common feature of being based on an optimized (and interruptible) flushing process under real-time conditions. The successive binary data blocks written to long-term memory during the flushing process follow variable, low-redundancy encodings adapted to the specific channels / sensors being recorded. Therefore, they are not intended to be self-describing.

[0012] Instead, the information required for decoding (hereinafter also referred to as interpretation) the samples is located elsewhere in the stored file. These locations, which are typically uncritical for a fast storage process, can be situated arbitrarily before, after, and possibly even in the middle of the binary information to be interpreted, depending on the recording method and instrumentation of the logger; sometimes, encoding components are also required in a separate signal catalog of the observed machine complex.

[0013] Compression techniques play a role in this context only within the data blocks and are adapted to the capabilities of the logger hardware. Run-length encoding and the Lempel-Ziv method are favored here. It is also expected that encryption methods will be used at this level of individual (compressed) data blocks in the future, since signal data typically contains sensitive information about the systems. Subsequent compression of already encrypted information has proven to be insecure and impractical.

[0014] In summary, this means that the simplicity and speed of recording in these formats comes at the expense of a significantly more complex interpretation, which typically only occurs after the measurement data has been transferred to a central host computer with connected random-access memory (RAM, SSD). The random-access memory allows for the alignment of nested binary data and its associated decoding instructions through arbitrary forward and backward jumps (random access). However, this approach is limited by a minimized overall host bandwidth and increased latency due to jumps and main memory requirements, particularly between the few processors (CPUs) and the random-access memory.

[0015] In connection with modern file and network systems, further formats have been developed that allow for simpler interpretation by using, for example, fixed encodings or self-describing data block structures, such as "CARMEN - Car Measurement Environment," "Extended Log File Format," or "Hierarchical Data Format." However, for the reasons mentioned, these play virtually no role in the time-critical recording of measurement data. This means that the necessary conversion of the recording formats into these intermediate formats inevitably involves a previously described, host-bound transcoding process (here: the sequential execution of interpretation / decoding, transformation / analysis, followed by recoding into a new target format).Since these intermediate formats – unlike the Big Data approaches described below – do not primarily help with the efficient analysis of the signals, their role currently remains limited to long-term archiving in the cloud.

[0016] To store and analyze huge, variably structured datasets in scalable computing environments, so-called Big Data methods have been used very successfully for several years. In this approach, a source file, for example, a very large text from which we might want to calculate the average sentence length per initial letter, is first divided into segments of roughly equal size and distributed among several general computing units using a network operation also known as segmentation or mapping. These computing units, with a few CPUs, some RAM, and sequentially operated hard disk storage, form a cost-effective yet very powerful computing cluster by working in parallel and independently of each other, for example, to identify sentence ends and initial letters as they process their respective text segments.

[0017] The prerequisite for this, however, is that the computing units for generating partial results already possess all the information for decoding the binary input data, either as a convention via the fixed character set or as a spontaneous, point-by-point exchange of technical information between any two cluster units via the network, e.g. by moving sentence ends, which were cut off by the technical segmentation of the original text, to the computer that also hosts and is to process the sentence beginning.

[0018] The final merging of the partial results across the computer network is attempted during a subsequent "shuffle" phase (sometimes also called a reduce phase). In this phase, the partial results (here: each event that identifies a measured sentence length with a specific initial letter; the initial letter is referred to as the event's shuffle key) are published to predefined units of the computer network responsible for a specific range of shuffle keys. There, they are then summarized in logically isolated segments of a "virtual" results file. "Virtual" in this context means that the relationship between the result segments distributed across the computer network (here: excerpts from the catalog of initial letters with their average sentence lengths) is only indirectly established, for example, through naming conventions or the creation of central index files.

[0019] When encoding the result segments or their index files, text formats or the relational analysis formats discussed below are usually used again, so that they can in turn be used directly as a starting point for further map-shuffle operations and thus for generating complex analyses from comparatively simple individual steps such as the described averaging.

[0020] Understanding this relationship can be aided by knowing that transformations, or the algebraic operators inherent in them, can generally be described in distributed systems as functions on relations of the form Δ: P R 1< × ... × P Rn< → P R'< describe it, whereby P R< ⊆ ( R ) represents the set of partitions of R, i.e. { P ∈ P R< ⇒ ∀ r ∈ R ∃! R ⊇ pEP mit r ∈ p}. Partitions correspond to the individual segments of the relation represented in the distributed system.

[0021] While the first generations of big data systems required static program logic, today's systems can be dynamically configured in algebraic query languages, following the tradition of "A Guide to the SQL standard: a user's guide to the standard database language SQL, 1997." The predefined analysis recipes are transferred via a selected computing unit of the cluster (driver) into a processing plan (graph), whose steps (nodes) are delegated to other computers (workers) and executed in parallel. These nodes consist of a set of basic operators (operator = implementation of a mathematical operation, i.e., a function), which, in addition to the "map" and "shuffle" operators already described, also include more specialized steps such as filtering, joining, sorting, and aggregating relational datasets.

[0022] These approaches reach their full potential when used with tailored relational analysis formats, such as "ORC - the smallest, fastest columnar storage for Hadoop workloads, 2018" or "Parquet - Big Data Concepts, Theories, and Applications, 2016". These formats employ, among other things, the compression and encoding strategies already mentioned to optimize storage requirements and analytical access. Due to their general nature, these approaches do offer technical redundancies (multiple storage of file segments) for load balancing and fault tolerance, but not logical redundancies (in this case, multiple storage of selected data records within the relation).

[0023] So far, it has not been convincingly possible to effectively utilize mobile and locally generated signal trace recordings with large data volumes, especially in the gigabyte or terabyte range, for such intelligent clustering methods. Even if it were possible to integrate the recording media of the logger units into the central cluster environments via broadband (a popular but highly inert approach in engineering is the physical transport of, for example, SSDs by mail, to then insert them into special network slots in the target cluster), the output formats of the measurement data, i.e., the signal trace recordings, are neither text-like files (with fixed encoding) nor fixed-structured artifacts (with encoding that can be read at a predictable location in the file).Technically segmented signal trace recordings are therefore not easily interpretable in parallel; therefore, an inefficient, host-based pre-transcoding into Big Data analysis formats is still generally practiced.

[0024] A second disadvantage of this approach is the display of samples from predefined channels. C ={ c i } as a relation R= {( t ,m ) = ( t,m 1 ,...,m n )|∀( c i ,t,m i )∈ σ ( t )} so to speak, as the only trace of news tuples MThis is because relational formats and their algebraic operators generally "know" too little about the signal domain, especially the temporal order and the associated analysis methods (example: Kalman filter (A New Approach to Linear Filtering and Prediction Problems, 1960)). Typically, such statistical analyses iteratively perform operations on all signals of a dedicated channel (channel track, used synonymously with channel below) and thereby generate new, computed channels. Filtering removes all signals whose messages rank in a suitable (lexicographical) order < below / above a reference message, or whose timestamp lies before / after a reference time in the temporal order <, or even complete signal traces from irrelevant channels. Derivation represents temporal message differences within a channel. Integration accumulates messages over the time course of a channel. Further accumulations include minimization, maximization, and averaging of signal traces. Events describe the (non-)presence of messages within a channel as linkable, Boolean messages from ( M b = { w,f } , & ,f ) .Resampling is the process of compressing or stretching a channel in terms of temporal order. Signal traces can also be shifted in time. Another essential group of operations, called combinations (the corresponding operators will be referred to as combinators in the following), encompasses the concatenation and subtraction of multiple signals and channels, each defined according to the respective message operations +̇ (correspondingly: ∸). In numerical domains, products, division, and exponentiation also exist as combinations. In Boolean domains, conjunction, disjunction, and negation can be defined analogously.

[0025] In general analysis formats, samples of different frequencies and channel compositions (corresponding to the recorded communication buses / formats) are now also stored in separate segments. The combination of relational tracks R i This is done via upsampling. U on a given interpolation routine Φ (so that all tracks behave as if they had been recorded at the highest frequency) with a subsequent join J (to combine the tracks of different formats). Φ : T × T × M ¯ × M ¯ → M ¯ U T' ⊇ T ( R ) = {( t',Φ ( t 1 , t 2 , m 1 , m 2 ))| t 1 ≤ t' ≤ t 2 minimal, so that ( t 1 , m 1) , ( t 2 , m 2 )∈ R} J ˜ R 1 , … , R n = t , m ¯ 1 , … , m ¯ i t m ¯ i ∈ U ∪ Ti R i J:PR1×…×PRn→∪J˜P1i,…,Pnj

[0026] This essential operator – even with advanced implementation using partition pruning (the early elimination of unnecessary partitions that diverge in time) and merge sort (the relations to be combined are first reordered according to a comparison key, such as the timestamp, to reduce interactions between the segment units) – has a multiplicative complexity and requires intermediate storage and re-partitioning (i.e., re-segmentation including network communication).

[0027] A third disadvantage of "classic" big data signal processing becomes apparent when an engineer wants to further process the analysis results with their familiar tools, for example, to play back the derived signal traces in a hardware-in-the-loop simulator (Electronics in Automotive Engineering, 2010). These tools are generally unable to handle the distributed analysis formats. For the sake of simplicity, host-based, sequential transcoding processes are therefore used again, which first transfer the segments to a central direct-access memory before they can ultimately be rewritten into the domain formats of, for example, the automotive sector.

[0028] AA Munshi and YAI Mohamed, "Cloud-based visual analytics for smart grids big data," 2016 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT), Minneapolis, MN, USA, 2016, pp. 1-5, doi: 10.1109 / ISGT.2016.7781163, describe how smart meters are a key technology for transmitting information between service providers and end users. However, the vast amounts of data generated by smart meters for visualization and control must be adequately managed to increase the reliability and sustainability of the smart grid. Interestingly, smart grids can be viewed as a big data challenge involving massive datasets and their analysis. Therefore, this unprecedented smart grid data requires an effective platform that brings the smart grid into the big data age.This paper presents a visual analysis system that can promote the sustainability of smart grids. One application of the framework was applied to a smart grid with over 6,000 smart meters to visualize dynamic demand management. Furthermore, another application is presented, using data from micro-generators, including an electric vehicle. The results indicate that the framework is suitable for performing visual and other analyses of smart grid data.

[0029] The object of the invention is therefore to provide a method and a device that make it possible to temporarily store and efficiently evaluate measurement data of several gigabytes, terabytes, and / or petabytes in size. This object is achieved by the features of the independent claims.

[0030] According to the invention, a method for processing measurement data using a computer unit arrangement is provided, wherein the measurement data is generated particularly during the operation of vehicles. At least three computer units (preferably each with its own CPU, indexing direct access memory, and sequential long-term memory) are provided for this purpose, wherein the computer units are connected to each other via a data link (e.g., a communication link – Local Area Network – LAN, such as Ethernet) by means of at least one switch. One of the computer units acts as a driver, performing a control function, while the at least two other computer units function as workers. The measurement data is provided to the computer unit arrangement via a data stream, and the measurement data typically has different data formats.In particular, dynamic transcoding requests and the transmission of abstract results (e.g., events) from the regionally stationed computing unit arrangement can be provided as a service via a global (mobile) communication network (generally: Internet).

[0031] The core of the invention is that the driver first analyzes the measurement data with regard to its different, especially variable, data formats ("jump scan") and thereby extracts structural and / or format information about this measurement data. From this structural and / or format information, the driver creates a processing and / or storage plan for the measurement data, particularly in the form of a table. This processing plan is optimized with regard to the different data formats, as it further divides the measurement data into suitable segments, which are assigned to the work units for parallel processing, especially for decoding. For this purpose, segments can technically overlap; that is, they do not represent pure partitions, but rather multi-sets. In particular, a part of an original file can thus be assigned to several workers, each with a different interpretation focus.The technical classification of measurement data and interpretation priorities is carried out in particular with regard to channel groups and / or time periods to be analyzed together.

[0032] This method, combined with a suitably selected number of worker units, enables efficient and rapid transmission and analysis of measurement data from local vehicles, even in the terabyte range or larger, within just a few minutes. This is because the initially unknown measurement data cannot be easily segmented and processed independently, as the resulting segments have different file formats, each with its own scattered decoding information. Furthermore, fragmenting measurement data originating from a specific source—for example, data collected by a particular sensor at different time periods—into different segments simply because it was stored at various points in the logger file for recording purposes proves inefficient and error-prone.According to the invention, by initially analyzing the structure and / or format information only in a rapid jump scan, a processing plan can be created in which the segments are optimized so that, for example, they only contain signal traces from a single predetermined source with a defined data format. This is because only largely functionally and technically independent segments can be processed efficiently in parallel by the work units.

[0033] The invention also encompasses the case where a segment can contain different file formats from logically and temporally related sensors / sources that are involved in a subsequent analysis (channel group). In any case, the processing plan must be created in such a way that the subsequent parallel processing by the work units can proceed as quickly, efficiently, and in a coordinated manner as possible.

[0034] Preferably, the processing plan therefore contains the structure and / or format information of the respective segments acquired during the jump scan and makes this information available to the processing units. This structure and / or format information also includes all instructions, especially decoding information, that the processing units need to process the segments. This enables efficient processing of the segments, as the individual processing units do not have to request the decoding information over the network each time or even read it from the measurement file via the data connection, which would disrupt the sequential transfer process. Instead, they access it from the processing plan stored in main memory.

[0035] Advantageously, the driver performs a jump scan when dealing with measurement data formats, especially those with changing formats, where the lengths of subsequent data blocks of a known structure can be predicted. This jump scan skips bytes of the data stream that fall within the predicted length. This allows for a very efficient and rapid pre-analysis of the original file or measurement data regarding its structural and / or format information, despite the underlying data volume, because the user data between these jumps or structural data is not transmitted at all. This means that the jump scan can be implemented very efficiently with sequential storage. If predicting the subsequent user data structure is only partially possible ("heuristic"), the format changes can be handled by combining forward jumps with as few backward jumps as possible within an iterative search procedure, with only minor drawbacks.to limit the sequential memory access.

[0036] The interpreted data segments processed by the work units therefore have one or more data channels, with each message or event / signal assigned a timestamp to one of these data channels. This enables analysis to determine when a specific sensor detected a particular event.

[0037] The preferred approach is to reduce the size of processed data segments while preserving information, using adaptive downsampling. This is achieved by having each processing unit locally cache only changes to message content, including the associated timestamps / periods, as a compressed, sequential segment file. This offers significant savings potential, especially for vehicle measurement data. This is due to the fact that messages in the data channels, such as recorded oil pressure, often maintain a constant value over extended periods (several engine revolutions with increasing speed). By adapting the data format to capture only changes to these values, the file size is efficiently reduced while preserving the chronological order.In practical applications, compression levels of up to 10% of the recording formats have been achieved, depending on the recording format and the signal change frequency. The intermediate storage format described in this invention can compete with even the strongest compression methods for recording formats, while simultaneously retaining the properties of full information preservation, segmentability, and parallel analyzability.

[0038] Preferably, the driver receives from the processing units, in response to its interpretation request, a directory of the compressed intermediate segments, the time intervals described therein, and further message statistics (smallest and largest messages; number of signals; average sample rate, replacement of frequently occurring messages with space-saving placeholders, ...), so that it can create a complete overview (index file) of the interpreted, now virtual, because distributed, measurement file.

[0039] This allows him to subsequently instruct the work units to exchange specific information at the channel- and time-related interfaces of the segments (time- and channel-based shuffle) to achieve effective later evaluation without further communication overhead. He can also ensure that the most important segments are not only sorted chronologically but, if necessary, also stored directly in the main memory of the work units to accelerate later analysis. Therefore, the index file can preferably be used to relate individual channels from sensors and their signal traces to each other via the resources of the distributed system. The index file, or...The compressed and chronologically sorted segment files referenced therein (hereinafter also referred to as "virtual analysis file") as a result of the interpretation stage (the first mode "mobile refueling" of the invention) thus represent an optimal basis for further analyses and transformations (the second mode "online transformation and (re-)synthesis" of the invention).

[0040] Preferably, information from the virtual analysis file is evaluated by executing the filtering, combination, derivation, and / or integration operations already described on the channels, applying logical conditions. This allows, for example, engineers to send declarative queries ("transformation recipes") to the interpreted measurement file or a driver computer via the internet. For instance, the query to determine when the values ​​of channel 1 are above value X1 and the values ​​of channel 2 are simultaneously below value X2 can be iteratively determined as a signal trace of a new, derived (Boolean) channel from the compressed message change signals for each segment / responsible work unit. These individual operations are calculated as operator nodes in a parallel analysis process initiated by the driver based on the index file and a specific analysis plan.The individual results (based on segments of the abstract signal changes in the truth value of the outlined Boolean condition) are preferentially cached in the memory of the participating SPDs as a "virtual measurement file" in order to be finally transmitted to the client as a data stream via the driver and the SAN. We will examine these efficient operations, which operate on such segmented signal changes rather than on signals themselves and whose complexity is therefore lower than the multiplicative methods already described, in more detail below.

[0041] According to a further aspect of the invention, the (mobile) intermediate storage device (signal processing cluster - SPC) for carrying out the methods described above is specified. The intermediate storage device has a first interface (SAN) for a data line – only temporarily connected during a maintenance procedure – to a vehicle in which the measurement data is generated. This measurement data is transmitted sequentially from the vehicle to the intermediate storage device via the first interface and with a very high bandwidth, preferably via a wired connection. Furthermore, the intermediate storage device has a computing unit arrangement that interprets the measurement data immediately after connection to the first interface and / or processes it according to a request, in particular from a cloud service provider, which is transmitted via a second interface (Internet).This second interface is intended for a second data connection to a public or private cloud network, whereby the analyzed measurement data or the results of the transformation requests formulated via this interface are forwarded to the cached measurement data via the second data connection. This data connection is considered permanent, but with varying and moderate bandwidth (compared to the first interface), depending on the situation.

[0042] Finally, a permanent data connection (intranet) between the units of the intermediate storage facility is preferably also provided.

[0043] The "refueling" takes place in close proximity to the regionally deployable intermediate storage unit. This allows regular maintenance operations or even routine refueling to be used to effectively buffer a vehicle's measurement data in the mobile intermediate storage unit. The intermediate storage unit closes a technical gap that exists because globally operating vehicles will generate several terabytes of local data per hour of driving in the future, but this data cannot be cost-effectively transmitted to a central server for further processing via existing networks. Conventionally, the data can therefore only be stored and transmitted in fragments, for example, by constantly overwriting the ring buffer in the vehicles, leading to information and data loss.

[0044] Preferably, the first data line can be implemented as a high-frequency cable connection, typically limited to a few meters of transmission distance due to interference, such as a fiber channel. The second, external data line has LAN or Wi-Fi connections. If the higher-bandwidth LAN connection is active, it is preferred. If, for example, no LAN connection is available due to the mobile unit's current location during refueling, the system switches to the Wi-Fi connection, and some bandwidth-dependent features of the unit (e.g., the size of the generated virtual measurement files) are restricted.

[0045] The Wi-Fi mode serves two main purposes. It can be used situationally when only small, but critical, amounts of data need to be transferred from the buffer to the cloud, for example, when anomalies occur in individual sensors. However, during the interpretation of the measurement data by the processing unit, it can also happen that the processing units lack necessary encoding or decoding information (for example, in the case of encrypted measurement data) and this information needs to be made temporarily accessible (e.g., via a one-time key or a vehicle-specific message catalog). The driver handles this task when creating the processing plan and can request the necessary information from associated security and catalog services via the second data line.

[0046] Preferred embodiments of the present invention are explained below with reference to the accompanying figure: Fig. 1: shows the intermediate storage device according to the invention for measurement data, in particular of a vehicle. Fig. 1b: shows the detailed structure of a single processing unit of the intermediate storage device for measurement data. Fig. 2: shows a first operating mode of the intermediate storage device. Fig. 1 For mobile refueling and interpretation of the measurement data. Fig. 3: shows a preliminary analysis of the measurement data for generating a processing plan. Fig. 3b: shows the nested binary structure of a measurement file, including format changes from practice in the Measurement Data Format (MDF). Fig. 4: shows the parallel interpretation and generation of a novel, compressed signal buffer format ("virtual analysis file"). Fig. 5: shows the second operating mode of the buffer device. Fig. 1 , also referred to as online transformation and (re-)synthesis, which processes the buffered measurement data for a client. Fig. 6: shows the iterative analysis or transformation of the compressed signal data using an example. Fig. 7: illustrates the completion and output of a (re-)synthesis job, which is processed according to the invention. Fig. 8: demonstrates an application of the invention or the interaction of several regional buffering facilities in the interactive analysis of globally distributed pre-series vehicle tests.

[0047] Numerous features of the present invention are explained in detail below with reference to preferred embodiments. The present disclosure is not limited to the specific combinations of features mentioned. Rather, the features mentioned here can be combined arbitrarily to form embodiments according to the invention, unless expressly excluded below.

[0048] Fig. 1 Figure 1 shows the intermediate storage device 100 according to the invention for measurement data, in particular from a vehicle. This intermediate storage device 100 can be described as a "data filling station," whereby, in contrast to a conventional filling station, in this case "something" is generated and delivered by the vehicle. The term "data filling station" is intended merely to clarify that the vehicle, particularly when "refueling" with electricity or gasoline or during a maintenance procedure, transfers the measurement data generated in it to the intermediate storage device 100 and can then be operated quickly and without data loss due to storage limitations. However, the invention also encompasses other intermediate storage devices 100 to which the vehicle can drive or which can be brought to the vehicle.The essential idea here, however, is that the intermediate storage device 100 can be brought so close to the vehicle that a first data line 105, in particular a data line 105 realized by Multi-Port Fiber Channel or other serial connections, is formed between the vehicle and the intermediate storage device 100, so that measurement data from the sequential memory 115 of the vehicle, which has been written by the logger 120, can be transmitted via a common first interface 110 to a computer unit arrangement 125 with a very high data rate, in particular in the range of several terabytes during a maintenance process lasting several minutes.

[0049] The intermediate storage device 100 is a novel combination of hardware and the data processing methods operating on it, described below, which closes a technical gap in the local interpretation and intermediate storage as well as the global, dynamic transformation and (re)synthesis of detailed signal trace data. Preferably, the measurement data is composed of or comprises such signal traces. "Dynamic" here means that the type and focus of the transformation requests, e.g., for the detection of events from specific signals, are not known in advance but change during operation due to business and economic motivations.

[0050] The hardware innovation lies particularly in a semi-stationary, scalable combination of modern network and storage network technologies paired with sequential signal processing units, which are described in detail below. Hereinafter, the storage network technologies will also be referred to as SAN (Storage Area Network). The novelty of the measurement data processing methods concerns two operating modes of the 100-unit buffer.

[0051] The first mode, which is described below, can be referred to as "mobile data refueling" of the intermediate storage device 100 and is characterized by a unique, distributed interpretation of the signal traces recorded in the measurement data with a highly optimized signal analysis format.

[0052] The second mode, which is also described below, can be called "online transformation and (re-)synthesis", using a newly developed refinement language whose time-period-associative operators calculate independent parts of a virtual signal output stream and which benefit from the cached signal analysis format.

[0053] Fig. 1 This shows that the connection to the logger 120 or the sequential memory 115 of the recording system or vehicle is via a multi-port SAN. A "Fiber Channel 128 GFC128" can bridge distances of up to 100 m at a raw transmission rate of 128 Gbit / s. Therefore, for single-channel data transmission from a fully measured vehicle to the intermediate storage device 100, the measurement data for one hour of driving (approximately one terabyte) can be transmitted in about one minute. Depending on the system and logger memory used, lower bandwidths or multi-channel connections can also be implemented. Each computer unit 135 is connected to the logger 120 in port-to-port mode via a switch 130, meaning each computer unit has its own data stream window into the collected measurement data. In the following, the computer units 135 will also be referred to as Signal Processing Devices 135, abbreviated as SPD.

[0054] The connection to a central server or a global cloud 170 is established via a second interface for a second data line 140. This LAN connection can be implemented either via Wi-Fi, particularly for brief queries to global services during mobile refueling, and / or via an internet-enabled leased line (several gigabits per second in stationary online mode). This is handled or implemented via a LAN switch 145, which also connects the computer units 135 to each other.

[0055] Depending on the application domain, at least two computing units in the buffer facility 100 are connected to the switches 130 and 145. The resulting signal processing cluster SPC 125 therefore consists of a virtually unlimited number of SPDs 135, which are configured according to Fig. 1b are equipped with their own CPUs 147. The actual signal data storage 150 of the computing units 135 is an inexpensive sequential storage medium, such as magnetic tape or a hard drive. All content information and statistics about the signal data storage, as well as intermediate results (reel files), are stored on a fast, small direct-access storage medium, e.g., an SSD 155. The minimum main memory 160 serves as a sample cache and is based on the typical block size (number of channels, number of channel groups, size of the analysis time windows) of the logged data of the operational domain. In the case shown, the four computing units 125 are not redundantly designed. Additional failover can be achieved by storing the data segments multiple times and using a suitable coordination protocol.

[0056] The buffer unit 100 can preferably operate in both modes simultaneously, provided the capacities of the described computing and network units are sufficient. Otherwise, a change in operating mode can occur upon the following events: When certain SAN ports are connected, the device switches to the first mode, "mobile refueling." If the connection is interrupted, the first mode is deactivated. The buffer unit 100 switches to the second mode, "online transformation and (re)synthesis," when the second data line 140 is activated via the leased line by a request. If this connection is interrupted, the second mode is deactivated.

[0057] Activating this mode means that all SPDs receive the respective event from LAN switch 145 or SAN switch 130 (broadcast). A coordination protocol is then activated, which identifies one of the SPDs 135 as the driver. Depending on the mode, the driver delegates the tasks or the function to be implemented to the remaining SPDs 135, which then act as workers.

[0058] Switching off the mode means that the driver receives the respective event or the result of the processing and then initiates the orderly termination of all ongoing calculations at the processing units. During processing, the driver typically also acts as a processing unit.

[0059] Fig. 2 Figure 1 shows the first operating mode, "mobile data refueling," of the intermediate storage unit 100 for processing the measurement data. For the execution of this first operating mode, it can be advantageous if the second data line 140 is only activated in narrowband mode via Wi-Fi, so that information for interpreting and decoding the measurement data 225 can be provided to the intermediate storage unit 100 via global services 170. This information includes temporary cryptographic keys or signal catalogs for the on-board electronics currently used in the vehicles, which then do not necessarily need to be pre-installed on the computer arrangement 125. Instead, this information can be requested from the intermediate storage unit 100 by a server, for example, with minimal redundancy, as needed. In this mode, the sequential memory 115 is connected to the intermediate storage unit 100 via the first interface 110.

[0060] First, as described above, one of the computer units 135 is selected as driver 135a. This driver 135a initially analyzes the measurement data only with regard to its different file formats, thereby extracting structural and / or format information about the measurement data. Since this processing step 205 is performed quickly and numerous data blocks of the measurement data are skipped, it is also referred to as a jump scan. To perform this jump scan, global key or catalog information may be required, which is either stored on the intermediate storage unit 100 or provided via the second data line 140.

[0061] The jump scan 205 is a method performed by driver 135a to quickly obtain structural information about a sequentially structured file, specifically measurement data. With this method, the actual user data is initially irrelevant; only those points within the measurement data where relevant changes in file format and / or decoding information are found are considered. Generally, these points can be quite arbitrary depending on the recording configuration. Therefore, there are two basic methods for performing the jump scan, depending on the available format.

[0062] The first method is referred to as "informed jump scan." Its distinguishing feature is that the rapid search for format changes within the measurement data is accelerated by providing the informed jump scan with information, particularly about the length of the data blocks in the various formats. The informed jump scan therefore works with formats where the lengths of the data blocks are announced before the user data, for example, via index blocks (MDF, MAT, MP4, AVI, etc.). In this case, the jump scan can use this knowledge to jump unidirectionally from one structural information point to the next by instructing the SAN connection to skip the already known length of subsequent user data.As a rule, this first step of "mobile refueling" reads less than one percent of the measurement data in order to subsequently create a parallel processing plan 210 for the actual user data and therefore only takes a few seconds.

[0063] The second method is called a "heuristic jump scan." This method uses format-dependent predictors for the length of the user data following a format change and thus performs a heuristic search (e.g., in the form of a binary search). In the middle of a segment under investigation, the current format hypothesis is checked, for example, by targeting suitable "magic bytes" (typical data arrangements for the respective format). Depending on the size of the segment and other factors, the probability that the format has changed between the beginning and middle of the segment is then determined. If successful, i.e., the probability is below a predefined threshold, the heuristic scan can continue the check with the last quarter of the file; otherwise, it rewinds via the SAN interface to the second quarter of the file.The more accurately the prediction / verification of the format hypothesis works, the fewer forward and (even fewer) backward jumps are needed to determine the format changes within the measurement data, ideally with logarithmic complexity.

[0064] Both methods 205 can thus provide information on where format changes occur in the measurement data (especially when switching from one signal track to a new one), and information can be extracted or requested (for example, via a central server) on how these formats can be processed or decoded. As a result, a processing plan 210, also referred to as an "interpretation plan," can be generated from the entire file, containing the required decoding information. This interpretation plan is several times smaller, typically less than one percent, than the original file, since its size depends primarily on the number of different formats, i.e., the different sensors and control units, and only to a small extent on the recording duration, which dominates the original measurement file.According to the interpretation plan 210, the available work units 135b are distributed segments of the measurement data of as equal size as possible for processing. Using the information from the processing plan, the work units 135b can now divide their portion of the total stream / measurement data 220 into individual samples or signals via their own SAN ports and forward them directly to a local storage method, which is described below. If necessary, both the driver 135a and the work units 135b can access global services via the LAN switch 145 and the WIFI interface 140, either to temporarily decrypt encrypted information or to obtain supplementary information about the channels and messages contained in the measurement data from an on-board network catalog.

[0065] The interpretation plan 210 can therefore present the structure or encoding information of the measurement data in tabular form. This tabular processing plan preferably indicates from which byte (position) and to which byte (length) within the measurement data a specific data format is present. These sections define the data windows or segments 220. Additionally, the interpretation plan can include a further column containing the decoding information for the respective segments. The interpretation plan can also contain a fourth column that restricts the channel or time information contained within the segment, resulting in technically overlapping segments that nevertheless correspond to the functional partitioning of the measurement file.The driver 135a now coordinates the parallel processing of the measurement data by the work units 135b using the interpretation plan 210, by providing the work units 135b with selected segments, including the respective decoding information.

[0066] This can be achieved, for example, by each work unit 135b connecting to its assigned segment 220 of the data stream / measurement data via a SAN port-to-port (P2P) connection 130 (map operation). The work unit 135b then interprets the binary data using the plan structure 210 transmitted by the driver 135a, or its corresponding excerpt thereof, and writes the results, encrypted or unencrypted depending on security requirements, into a novel signal analysis format in its respective signal data storage 150. The resulting metadata about the interpreted channels, signal and time values, and storage locations are initially collected in the main memory 160 of the work units and, after successful processing, reported back to the driver 135a, which then coordinates a central index file 237.

[0067] The driver can, among other things, cause the processing units to exchange information with each other via specific channels and / or time periods. While in the general shuffle operation all data records are potentially exchanged between all computing units via the network, the time-period shuffle 236 consists only of the targeted communication of a small subset of the data records at the edges of the time periods for each processing unit to a single, dedicated target unit that manages the respective connecting period. Finally, the resulting index file 237 is redundantly stored on all index stores 155, so that from now on the role of the driver for the subsequent analysis (see the following explanation of the "online transformation and (re)synthesis" mode) can be freely changed between the processing units.

[0068] Fig. 3 Figure 1 illustrates the jump scan of the measurement data for generating the processing plan. The starting point is the original measurement data file 200. The measurement file contains several signal trace pieces of information in sequentially written blocks, which are usually generated by successive flush operations from the logger's ring buffer. The format-homogeneous blocks themselves preferably contain a sequence of coded samples, which were built up by chronological observation processes in the logger's ring buffer. Via the Port-2-Port SAN interface 130, the driver 135a accesses the structured measurement file 200 and performs a jump scan 205 – either informed or heuristic, depending on the file type. From this, a processing plan 210 is derived and delegated via LAN to the processing units 135b for parallel processing 215.Via the P2P SAN interface, the individual work units 135b now also have access to the measurement file 200, but limited to the previously identified segments 220 that were assigned to them according to the processing plan 210. When interpreting the segments 220 or the blocks they contain, the work units 135b can access global services 225, e.g., to obtain cryptographic keys or signal catalogs. Initially, channel group-based sample structures 230 can be reconstructed from the blocks, preferably in main memory 160 and / or index memory 155, which are then further compressed and stored in parallel by the work units 135b 240. The original measurement data were preferably recorded sequentially and can therefore be easily arranged chronologically in representation 130.

[0069] Fig. 4 Figure 2 illustrates the subsequent process of generating the signal analysis format 250, whereby distributed, yet comprehensively indexed storage (we refer to this as virtual storage) of the signal traces is performed through adaptive downsampling. As already described in Figure 250, the following steps are described in Figure 250: Fig. 3 are also in Fig. 4 The initially decoded segments 230, also referred to as sample stream 230, generated by worker units 135b, are shown. The format of these sample streams 230 is based on the layout of the interpreted track segments. Typically, consecutive samples from a control unit bus (i.e., a channel group 1 that is technically linked due to shared communication) are followed by a set of overlapping samples from another recording source (channel group 2). Meanwhile, a completely different (temporal and logical) section of the original file can be processed on another worker 135b. These samples are initially wound onto disk 155 until all recording sources have been processed and a new time period begins.

[0070] Preferably, while interpretation 215 or the processing of the next set of samples 230 begins (i.e., multi-threaded), a further processing step, referred to as adaptive down-sampling 240, is started. In this processing step, the reel segments of all sources are simultaneously cleared from beginning to end and further processed.

[0071] Adaptive downsampling works as follows: first, the channels represented together in the samples are separated, and only the changes in their message content (signals / values) are transmitted. A change in message content would be, for example, if the content of channel 1 changes from "0" to "1" at a time of 10 ms. The sequence of these messages for a channel is also called the signal trace. The complete signal trace of channel 1 over a 10 ms interval is: 0, 1, 1, 0, 0.

[0072] The main memory 150 serves as a cache for message comparisons, and its size is at least linearly proportional to the number of channels or message sizes to be processed. Mathematically, the compressed representation of the signal traces corresponds to the following relationship: Q : c t 1 t 2 m ∀ t 1 ≤ t ≤ t 2 c t m ∈ σ t

[0073] The compressed representation of channel 1 was therefore 0[0-10],1[10-30],0[30-50]. The signal tracks from different sources (different channels, different sample frequencies) are mixed together simultaneously and written to sequential target files on memory 160, sorted according to the start time of the message change. The target files / record structures are tailored to the length information of the messages, i.e., known in advance. At the same time, statistics and content references on local disk 155 are regularly updated. Worker 135b thus generates a directory of its files, channels, and especially the time intervals for which it is responsible, and can also communicate this to Driver 135a during the interpretation process.Driver 135a can then very quickly generate an overall overview 237 (interval responsibility plan) and make it available to work units 135b for long-term storage and analysis on disk 155. In the final implementation of 137, several further key figures and statistics (number of samples, sample rate, marginal values, etc.) can be recorded, which support the subsequent analysis steps.

[0074] The entries of the signal traces at the beginning and end of the interpreted and stored time periods are now potentially relevant in analytical contexts for other work units (workers) with overlapping / adjacent responsibilities, e.g., to calculate derivatives. According to the updated overall view, precisely these entries, initiated by driver 135a, can be exchanged via targeted network communication (time period shuffle 236) between any two work units 135b (end signal channel X period 1 <-> start signal channel X period 2). In practice, this communication can be implemented on ( n - 1) * k limit message pairs, whereby n the number of segmented time periods and k represents the number of channel groups.

[0075] The incorporation of such received information occurs on the already written file through targeted, byte-wise overwriting. Because of the fixed storage format, targeted zero bytes can be added to the beginning and end of the file during the initial write operation; these are then replaced during the process. Consequently, individual track entries are stored redundantly in the final storage format, ensuring that the local data segments are temporally and logically self-contained and minimizing the necessary network communication during further analyses.

[0076] Tests with real measurement data from vehicle testing show compression rates for the novel signal analysis format using adaptive downsampling between 10% and 90% of the original file, depending on the dynamics of the recorded signals and any compression method already used during recording. By representing signal changes instead of samples, subsequent analysis saves up to 80% of the computing resources (CPU, RAM) compared to standard transformation operators.

[0077] The formats 230, 237 and 250 can be combined with centrally authorized encryption methods 225 when writing, for example to prevent data theft from the data refueling station independently of hardware cryptography components or to effectively separate the data of different clients from each other.

[0078] An exemplary result of a compressed signal track 250 is in Fig. 4 The individual channels are summarized in column 250a. Column 250b marks the start time and column 250c the end time of a message change for the respective channel, with the time units preferably based on the smallest available sample rate of all available channels (e.g., the number of microseconds since January 1, 1970), so that the individual channels can be related to each other in time. Finally, column 250d shows the content of the message as it appears upon a change.

[0079] The segments in analysis format 250, distributed across the memory 150 of the individual work units, together with the index data 237 redundantly stored in the direct access memory 155 on all work units, form a virtual analysis file in their logical totality.

[0080] Fig. 5 This section provides an overview of the processing, also referred to as transformation and (re)synthesis, of the analysis data for a cloud client 170, i.e., a user interaction or an automated process that requires targeted excerpts or abstractions of the raw measurement data. Preferably, the second data line 140 is active with high bandwidth for this purpose, enabling global services to be requested and customer requests to be fulfilled with the desired measurement files. The driver 135a provides a query service (endpoint) on an exported network port using a suitable internet protocol (e.g., HTTP according to the REST paradigm), which can be used by clients. A (re)synthesis job 260, for example, from a control center, a server, or even individual data-processing big data processes, is submitted to this service at the request of a client using a high-level signal transformation language (which is described below).In this high-level language, the client can formulate and request 1) the finding of events / message patterns, 2) the aggregation of signal / channel statistics, 3) the export of raw signals, and / or 4) the derivation of artificial signals (calculated from measurement channels). Within the framework of high-level language 260, client 170 can also transmit temporary authorization or cryptographic keys if the data refueling station is configured for multiple clients and / or the underlying analysis files are encrypted.

[0081] The driver 135a translates the high-level language using the index data 237 into a distributed processing graph for signal transformations, the so-called analysis and synthesis plan 270, which is transmitted and coordinated via the LAN interface 145 to the workers 135b for parallel execution. The nodes / operators 280 of this plan 270 are divided into two categories. First, there are novel, period-associative refinement operators, which are implemented as an iterative chain, i.e., acyclic, based on the signal analysis format 250 described above and the time-period shuffle 236 already described. The intermediate results or segments 285 of this operator chain are wound onto the index memories 155.

[0082] The final nodes of the operator chains (270) contain output instructions, including the desired encodings for the target formats (310) requested by the client (170). These output operators are, so to speak, the counterparts to the interpretation operators in mobile refueling, because the possible target formats include not only self-describing data types such as CSV, JSON, AVRO, etc., but also, and especially, the measurement data recording formats standardized in the industry. In particular, these formats can be optimized for the purpose of parallel synthesis so that format changes or data block lengths coincide exactly with the result segments (290) generated by the output operators on the signal storage (160). The final logical result (310) can thus be described as a simple concatenation of the result segments.For the purpose of optimising storage and network utilization, this concatenation is only carried out improperly in our invention by the driver 135a constructing a virtual output stream 300 for the client via a forwarding function of the SAN switch 130 to the respective work units or their signal storage 160, so that the client can also read the file 310 segment by segment and fault-tolerantly via the second data line 140.

[0083] Fig. 6 The refinement or (re-)synthesis of the analysis data is explained in detail using an example. At this point, it is important to reiterate the invariant that, in the described signal analysis format, all information belonging to a time period / analysis context is located within the corresponding segment 250, regardless of channel, source, and recording rate. Mathematically, such a structure can be described as an interval partitioning. Idescribe as a multi-set over Q I Q ⊆ P Q I Q ∋ I = i t 1 , i t 2 , … , i tn − 1 , t n ⊆ Q t 1 ≤ t 2 ≤ … ≤ t n ∧ ∀ c t a t b m ∈ Q mit t a ≤ t j , t i ≤ t b ⇒ c t a t b m ∈ i ti , tj .

[0084] This implies that almost all operations (combination, filtering, differentiation, event detection – except for integration, which is described below as an example of further, cross-period operations / accumulations) can be performed by local calculations 283 and entirely without resampling and re-partitioning. An example is shown in Fig. 6 for the concatenation of messages (i.e., addition of signals / channels) □ shown: + ˙ c 1 , c 2 : I Q → I Q ′ , + ˙ c 1 , c 2 I = ∪ + ˙ c 1 , c 2 i ∈ I + ˙ c 1 , c 2 i = c 1 + ˙ c 2 , t a , t b , m 1 + ˙ m 2 ∀ t a ≤ t ≤ t b ∃ c 1 t 1 t 2 m 1 , c 2 t 3 t 4 m 2 ∈ i , so that max( t 1 , t 3) ≤ t ≤ min( t 2 , t 4)}

[0085] Filtering / extracting individual channels is even more trivial. ← c 1 : I Q → I Q ′ , ← c 1 I = ∪ ← c 1 i ∈ I , ← c 1 i = c 1 , t a , t b , m ∈ i

[0086] The filtering of time references or entire periods is as follows: ↓ t1 ,t2 : I Q → I Q ′ , ↓ t1 ,t2 I = ∪ ↓ t1 ,t2 i ∈ I ↓ t 1 , t2 i = c t a t b m ∀ t a ≤ t ≤ t b ∃ c 1 t 3 t 4 m ∈ i ∧ t 1 ≤ t 4 ∧ t 3 ≤ t 2 , so that max t 1 t 3 ≤ t ≤ min t 2 t 4

[0087] Filtering a channel based on the truth value of another (boolean) channel is defined as follows: ↑ c 1 , c 2 : I Q → I Q ′ , ↑ c 1 , c 2 I = U ↑ c 1 , c 2 i ∈ I ↑ c 1 , c 2 i = c 1 t a t b m ∀ t a ≤ t ≤ t b ∃ c 1 t 1 t 2 m , c 2 t 3 t 4 t ∈ i , so that max t 1 t 3 ≤ t ≤ min t 2 t 4

[0088] The calculation of a Boolean channel, e.g., from the exceedance of a value of a numeric channel, is defined as follows: > c , m1 : I Q → I Q ′ , > c , m1 I = U > c ,m 1 i ∈ I > c , m1 i = c t a t b b ∀ t a ≤ t ≤ t b ∃ c t 1 t 2 m ∈ i with b = t ∧ m > ˙ m 1 oder b = t ∧ m ≤ ˙ m 1 , so that max t 1 t 3 ≤ t ≤ min t 2 t 4

[0089] For the sake of completeness, descriptions of further local refinement operators 283 can be found in the addendum to this document. The procedure is similarly resource-efficient as adaptive downsampling 240: The sorted input segment is processed from beginning to end. The cache 160 contains the current message of the input channels and the last calculated message of the output channels. The next input data record is read in, and if necessary, the cache is updated and combined into a new output message. Only when the input segment changes or ends is the next target record of the respective output channel appended to a new intermediate result segment 285 on the index memory 155. This means that the format 250 is retained for the next operator iteration of the processing chain 270.

[0090] The refinement operation of the derivative as a point difference (of course, more "continuous" operators can also be implemented according to our invention description, see Addendum) works as follows, taking into account the fact that the boundaries of the represented time periods are stored multiple times or redundantly: ∂ c i = ∂ c , t a , t b , m ′ c t 1 t 2 m 1 , c t 3 t 4 m 2 ∈ i mit t 2 = t 3 t a = t 3 , t b = t 3 + 1 , m ′ = m 2 ∸ m 1 oder t a = t 3 + 1 , t b = t 4 , m ′ = 0 ˙

[0091] Only in the case of integration—representing other, analogous or, in most cases, even simpler cross-period accumulations (see Addendum)—is the local information insufficient for calculating the time-accumulated messages. Nevertheless, such operations can be implemented associatively, meaning they can be divided into two phases: a comprehensive, work-unit-local calculation phase and a minimal coordination phase between work units mediated via targeted network communication.

[0092] In the case of integration, the segment integrals are first accumulated in the local phase: ∫ c i = ∫ c , t 2 n − 1 , t 2 n , Σ m j c t 1 t 2 m 1 , … , c t 2 n − 1 t 2 n m n ∈ i mit t 2 j = t 2 j + 1 <mprescripts / > <none / > ∼ Σ c ( i ) =m, so that ( c,t 1 ,t 2 ,m ) ∈ c ( i ) and t 1 maximum.

[0093] In the coordination phase, the last segment integral Σ is now calculated. c ( i ) in a variant of the time-period shuffle 236, this is communicated to all work units 135b operating subsequently in the chronological sequence. These workers 135b then adjust their locally accumulated partial result, which has already been written to the index memory 155, using the overwrite procedure already described (hereinafter by the overloaded +̇ : I Q< × M □ I Q< (Operation represents) again. So let I Q< ∋ I= { i 1 ,..., i n} an interval partitioning with n successive segments i j , then the following applies ∫ c I = ∫ c i 1 <mprescripts / > <none / > ∼ ∪ ∫ c i 2 + ˙ Σ c i 1 <mprescripts / > <none / > ∼ ∪ … ∪ ∫ c i n + ˙ Σ j < n Σ c ij <mprescripts / > <none / > ∼

[0094] The complexity of the coordination phase of the integration operator is therefore independent of the signal or time length of the underlying measurement file. n ! Further (inter-period) refinement operators 283, such as the time shift, which all fall within this complexity class and together form the basis of a suitable high-level analysis language for signal traces, are described in the addendum.

[0095] Fig. 7 illustrates the distributed, dynamic (re-)synthesis based on such a high-level language or query protocol 260, which aims to calculate targeted excerpts and abstractions from the originally recorded signal traces in several target formats (virtual measurement file 310) upon request from a global service / client 170: bool := 't'|'f' literal := '\'[A-Za-z0-9\:]* '\' number := -?(\d+(\.\d*)?|\d*\.\d+) ([eE] [+-]?\d+)? channel := (? <bus>[A-Za-z0-9\: / %]+)\.(? <name>[A-Za-z\_%]+)? operation := 'abs' | 'coalesce' | 'integrate' | 'derivate' | 'max' | 'min' | 'avg' | 'mean' | 'first' | 'last' | 'shift' | '!' operationCall := operation '(' formula (, formula)* ')' operand := '+' | '-' | '*' | ' / ' | '^' | '==' | '<>' | '>' | '< ' | '>=' | '<=' | '<' | '&' | '|' term := formula operand formula formula := bool | literal | number | channel | operationCall | '(' term ')' format := 'json' | 'csv' | 'mat' | 'vts' | 'dat' | 'mdf' | 'mf4' | 'atfx' trace_id := UUID job := 'when' formula 'in' trace_id 'then' format formula (, formula)*

[0096] The driver 135a translates this high-level language using the index files 237 (the virtual measurement files are implemented via suitable identifiers, here without restriction of generality as Universally Unique Identifier - UUID) into an acyclic processing graph, the analysis and synthesis plan 270, of parallel local refinement operators 283, synchronizing time period shuffle coordination nodes 236 and the final, again parallel, output operators 287.

[0097] The output operators convert the stream of message changes 285 generated by the preceding operations (per unit of work: time-partial), which is wound onto the index memory 155, directly into corresponding segments 290 of a desired target format (see also Fig. 6 ). Here, the characteristics of these target formats, such as optimal, redundancy-free encoding through format changes and compression, are exploited, and specifications from the (re)synthesis query are taken into account. For this purpose, the analysis and synthesis plan 270 contains precisely such "conventions" that apply for the duration of the query, so that the parallel and independently generated segments 290 predictably and seamlessly fit together into a logical overall file 310. The driver can therefore construct a forwarding data stream 300 even during the runtime of the transformation, which uniquely maps the data from the individual result segments 290 via the SAN interface 130 into a resulting measurement file 310 manifested via interface 140. This is achieved through the support of suitable protocol instructions, such as those used when sending media files, such as audio and video streams, e.g.HTTP streaming allows for globally interactive and distributed analysis processes to be designed using multiple regional data refueling stations in "online transformation and (re)synthesis" mode.

[0098] Fig. 8 explains the invention and its properties once again using a concrete case study that can occur in reality, but by no means limits the invention to this example.

[0099] In modern motor vehicles, especially passenger cars, a multitude of processes are electronically controlled and monitored. To improve passenger cars or to detect faults, this data must be evaluated regularly and as comprehensively as possible. For this purpose, vehicle manufacturers send fleets of specially instrumented pre-production vehicles to regions of the world with diverse climates and conditions. One problem is that such a passenger car can generate approximately one terabyte of data within an hour. If all the passenger cars were to attempt to transmit this data wirelessly to a central server, it is expected that this would overload the technical infrastructure of the regions or the vehicle manufacturer and / or result in significant costs. According to the invention, the prototype passenger car can now transmit the measurement data generated within it, for example, when it needs refueling, according to... Fig. 1 The data is transmitted to the intermediate storage facility 100 located in the respective regions. Due to the physical proximity to this intermediate storage facility 100, i.e., the data refueling station, a wired transmission can take place, enabling high data transmission rates so that the prototype can quickly resume its test drives. In this way, the sequential memory 115 of the car can be regularly emptied, so that no or only a small amount of the measurement data valuable to the manufacturer is lost, and at the same time the vehicle tests before series production can be completed on schedule.

[0100] The various sensors in cars typically write their respective measurement data to sequential memory 115 in different formats, meaning that a conventional big data analysis method, such as those often already used on text files in the automotive manufacturer's data centers, is not directly applicable. Text files, even within large original files, have a uniform format (character code) and can therefore be easily broken down into individual segments for parallel processing. Let's assume the measurement data contains signal traces from sensors that sequentially write the engine oil pressure, the selected gear, and the speedometer reading to memory 115 at different frequencies. The jump scan method according to... Fig. 3 This enables these three different frequencies and formats to be recognized and each delegated to work units 135b for parallel interpretation. The interpreted measurement data are stored in the intermediate storage device 100 in segments 235, which according to Fig. 4 The signals are subjected to adaptive downsampling and shuffled periodically, so that only the changes of, for example, the three signal tracks to be evaluated together are available chronologically. The intermediate storage device 100 can be scaled so that the measurement data of an entire regional fleet segment can be stored and analyzed in this compressed form, without having to collect it at a central location (the "bottleneck") in the vehicle manufacturer's data center.

[0101] An engineer at the vehicle manufacturer's headquarters, who wants to analyze the measurement data from the test fleet, can now, for example, send a request 260 via a browser client 400 to a central federation service 170 in his employer's cloud. This is because he is initially unaware of, and ultimately indifferent to, the location of each prototype in the world. He is only interested in the answer to the question, "At what speed and in which gear did the oil pressure rise above value XY?". As in Fig. 8 As illustrated, the federation service can forward the declarative transformation 260 ("when motor.oil_pressure>XY in fleet20 then csv gear.level, vehicle.velocity") to the respective intermediate storage facilities 100 in the test regions via the data lines 140 (where the fleet identification is replaced, for example, by the identification of the vehicles stationed in the region). Simultaneously, the federation service combines the regionally calculated virtual result files 310 (here: lines of the CSV format) into a final data stream during their creation, which manifests itself in the browser client 400 and allows the engineer to quickly gain insights into improving and preventing errors in the prototype vehicles. Insofar as the problems involve the control unit software, such improvements can even be deployed to the prototype vehicles via the vehicle manufacturer's infrastructure in a timely manner and remotely maintained, and monitored in close succession. Addendum

[0102] The difference operation (analogous to product, division, exponentiation) on multi-sets is defined as follows: ∸ c 1 , c 2 : I Q → I Q ′ , ∸ c 1 , c 2 I = ∪ ∸ c 1 , c 2 i ∈ I ∸ c 1 , c 2 i = c 1 ∸ c 2 , t a , t b , m 1 ∸ m 2 ∀ t a ≤ t ≤ t b ∃ c 1 t 1 t 2 m 1 , c 2 t 3 t 4 m 2 ∈ i , so that max t 1 t 3 ≤ t ≤ min t 2 t 4

[0103] Constant messages or artificial static channels can be represented as follows: K m : I Q → I Q ′ , K m I = ∪ K m i ∈ I K m i = m t a t b m ∀ t a ≤ t ≤ t b ∃ c 1 t 1 t 2 m 1 ∈ i , so dass max t 1 t a ≤ t ≤ min t 2 t b

[0104] The Boolean conjunction (analogous: discunction) is defined as: & ˙ c 1 , c 2 : I Q → I Q ′ , & ˙ c 1 , c 2 I = ∪ & ˙ c 1 , c 2 i ∈ I & ˙ c 1 , c 2 i = c 1 & c 2 , t a , t b , b ∀ t a ≤ t ≤ t b ∃ c 1 t 1 t 2 b 1 , c 2 t 3 t 4 b 2 ∈ i with b = t Λ b 1 = t Λ b 2 = t or b = f Λ ( b 1 = f V b 2 = f), so that max( t 1 , t 3) ≤ t ≤ min( t 2 , t 4)}

[0105] Negation completes the expressiveness on Boolean signal traces: ! ˙ c : I Q → I Q ′ , ! ˙ c I = ∪ ! ˙ c i ∈ I □(□)={(!□,□ □ ,□ □ , b )| (□,□ a ,□ b , b 1) □ □ and b = t Λ b 1 = f or b = f Λ b 1 = t }

[0106] A non-point-based variant of the derivation can be obtained by relating the message differences to the duration of the time or change periods: ∂ ˜ c i = ∂ ˜ c , t a , t b , m ′ ∀ t a ≤ t ≤ t b ∃ c t 1 t 2 m 1 , c t 3 t 4 m 2 ∈ i with t 2 =t 3 and m ′ = m 2 ∸ m 1 t 2 − t 1 , so that max( t 1 , t 3) ≤ t ≤ min( t 2 , t 4)}

[0107] Similarly, a time-sensitive integration operation can be built: ∫ ′ c i = ∫ ′ c , t 2 n − 1 , t 2 n , Σ ˙ mj ∗ t 2 n − t 1 c t 1 t 2 m 1 , … , c t 2 n − 1 t 2 n m n <mprescripts / > <none / > ∼ ∈ i mit t 2 j = t 2 j + 1 Σ ′ c i = m , so dass c t 1 t 2 m ∈ ∫ ′ c i und t 1 maximal . <mprescripts / > <none / > ∼ ∫ ′ c I = ∫ ′ c i 1 <mprescripts / > <none / > ∼ ∪ ∫ ′ c i 2 <mprescripts / > <none / > ∼ + ˙ Σ ′ c i 1 ∪ … ∪ ∫ ′ c i n <mprescripts / > <none / > ∼ + ˙ Σ j < n Σ ′ c ij

[0108] Maxima (minima, averages) are calculated using the following scheme: Max ˜ c i = Max c , t 2 n − 1 , t 2 n , m ′ c t 1 t 2 m 1 , … , c t 2 n − 1 t 2 n m n ∈ i with t 2 j = t 2 j + 1 und m ′ = max m 1 , … , m n Max c ( i )= m , so that c t 1 t 2 m ∈ Max ˜ c i and t 1 maximum. Max c I = Max ˜ c i 1 ∪ max Max ˜ c i 2 , Max c i 1 ∪ … ∪ max Max ˜ c i n , Max c i 1 , … , Max c i n − 1 .

[0109] Finally, we should also mention the time shift into the future (and symmetrically, the time shift into the past). In this case, the "overflow" portions of the shifted partitions are communicated to the subsequent work units / signal storage. → ˜ t i = c , t 1 + t , t 2 + t , m c t 1 t 2 m ∈ i → ˜ t + i <mprescripts / > t <none / > = c t a t b m ∈ → ˜ t i ∀ □ □ ≤ □ ≤ □ □ ∃ c 1 t 1 t 2 m 1 ∈ i , so that max(□ 1 , □ a ) ≤ □ ≤ min(□ 2 ,□ b )} → ˜ t − i <mprescripts / > t <none / > = → ˜ t i / → ˜ t + i <mprescripts / > t <none / > → t I = → ˜ t + t i 1 ∪ → ˜ t + t i 2 ∪ → ˜ t − i 1 <mprescripts / > t <none / > ∪ … ∪ → ˜ t t i n ∪ → ˜ t − i n − 1 <mprescripts / > t <none / > < / name> < / bus>

Claims

1. A method for processing measurement data by means of a computing unit arrangement, wherein at least a number of three computing units (135) is provided and the computing units are in data connection by means of at least one switch (130, 145), wherein one of the computing units performs a control function as a driver (135a) and the at least two other computing units (135b) operate as work units (135b), wherein the computing unit arrangement (125) receives the measurement data (200) via a data stream and the measurement data (200) is in different data formats, characterized in that the driver (135a) analyzes the measurement data (200) in relation to their different data formats and thereby extracts structural and / or format information from the measurement data (200), the driver (135a) creates a processing plan (210) for the measurement data (200) based on the structural and / or format information, wherein the processing plan (210) is optimized in relation to the different data formats, the measurement data (200) are divided into segments (220) according to the processing plan (210) and assigned to the work units (135b) for parallel processing.

2. The method according to claim 1, characterized in that the processing plan contains the structural and / or format information of the respective segments (220) and provides this information to the work units.

3. The method according to any one of the preceding claims, characterized in that the driver (135a) performs a jump scan, particularly when the formats of the measurement data (200) change, wherein the jump scan in this case can be unidirectional or bidirectional, skipping bytes within a known or predicted data payload length where no format changes occur.

4. The method according to any one of the preceding claims, characterized in that the interpreted data segments (230) have one or more data channels, wherein each message or each event of one of the data channels is assigned a timestamp.

5. The method according to claim 4, characterized in that the interpreted data segments (230) are compressed such that each work unit saves only changes in the content of messages, including the assigned timestamp, as a compressed segment file (250) .

6. The method according to claim 5, characterized in that the driver (135a) receives from the work unit (135b) a directory of channel groups and time intervals of the compressed segments that the work unit (135b) has processed, whereupon an optimized directory (237) and the integration (236) of the segment files into a complete analysis format are coordinated into a virtual analysis file.

7. The method according to claim 6, characterized in that information in the virtual analysis file is evaluated by applying refinement and output operators on the channels, in particular filtering, combination, derivation, and / or integration operations using logical conditions and their results are made available as a virtual measurement file.

8. A buffer device for performing one of the methods described above according to any one of claims 1 to 7, comprising a first interface (110) for a first data line (105) to a vehicle in which the measurement data (200) are generated, wherein these measurement data (200) are transmitted from the vehicle to the buffer device (100) via the first interface (110), a computing unit arrangement (125) that processes the measurement data and / or prepares them based on a request, in particular from a service provider, a second interface for a second data line (140) to a central server, wherein the processed measurement data (200) are transmitted to the central server via the second data line (140).

9. The buffer device according to claim 8, characterized in that the first data line (105) and / or the second data line (140) is configured as a WIFI and / or as a LAN connection.

10. The buffer device according to any one of claims 8 to 9, characterized in that necessary encoding and / or decoding information for processing the measurement data (200) can be queried and transmitted via the second data line (140) from the computing unit arrangement (135) of the buffer device (100).