SYSTEM FOR PARALLEL BUFFING AND PREPROCESSING DATA FLOWS BASED ON A MULTI-LEVEL CONTEXT IN VEHICLE DOMAINS

Abstract

System (100) for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains, comprising: a first data processing level (104) which includes a first-level buffer (106) designed to buffer data provided from a plurality of raw data streams (102a to 102d) output by a plurality of vehicles (101a to 101d);and a second data processing level (108) comprising an electronic task queuing directory (TQD) module and a plurality of second-level data processing buffers, wherein the TQD module is designed to create a plurality of tasks in response to receiving a serial data stream output from the first-level buffer (106), the TQD module assigning each task to a corresponding second-level buffer and splitting the serial data stream into individual data values ​​which are provided to a specific second-level buffer (108) based on the task.

Description

BACKGROUNDThe present invention relates to location-based services (LBS) and, more particularly, to systems for processing stream transmissions in vehicle domains.The increasing development of vehicle telematics, sensing and mobility technologies has resulted in a wide spread of mobile computing systems and location-based services (LBS). To improve vehicular telematics applications, it is usually necessary to transmit data of a moving vehicle with LBS in real time in the data stream. Examples of these enhanced vehicle telematics applications include online event notifications, congestion monitoring, real-time route guidance, approaching driver notifications for a ride sharing application, and autonomous navigation.Conventional attempts to process data stream transmission to support LBS applications include implementing existing internet-based frameworks such as Hadoop, Store, Spark, etc. However, these internet-based frameworks present various special challenges. For example, unlike conventional Internet-based computing methods that typically receive data from a fixed or substantially fixed terminal or node, the vehicle domain provides large sets of constantly moving terminals that operate under different conditions. Moreover, conventional GPS / 3G / 4G signals are unstable and are easily affected by numerous factors. For example, weather conditions, tall buildings, high roads, tunnels, bridges, etc. may contribute to signal faults, interruptions, and delays.In addition, Internet-based frameworks are usually designed to process independent requests. However, the vehicle domain differs therefrom in that requirements are not independent. That is, a single vehicle may make a variety of requirements on a single occasion, for example, contextual map and road network data, events, traffic / weather conditions, dynamic navigation instructions, and diversion descriptions. Thus, access to fast and large shared memory is required for parallel processing of requests to avoid providing legacy data to the end user.In this context, already published documents exist. Document JP 2000-183961 A describes a packet communication priority control procedure for use in a router. In this case, communication packets with low priority between those with high priority are processed. Further, document JP 5 303 593 B2 describes a communication packet control apparatus having a hardware interrupt processing circuit and a software interrupt time allocation mechanism for interface queues which switches between tasks.Despite these already achieved advances, there remains a need to optimize packet transmissions in networks to avoid providing legacy data packets to an end user.SUMMARYThis object is achieved by the subject matters of the independent claims. Further embodiments are evident from the respective dependent patent claims. According to an embodiment of the present invention, a system for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains includes a first data processing level and a second data processing level. The first data processing layer includes a first layer buffer configured to buffer data provided from a plurality of raw data streams output by a plurality of vehicles. The second level of data processing includes a task-queue directory electronic module (TQD) and a plurality of second level data processing buffers. The TQD module is configured to perform a plurality of tasks in response to receiving a serial data stream output from the first level buffer. The TQD module is further configured to assign each task to a corresponding second-level buffer and separate the serial data stream into individual data values provided to a specific second-level buffer based on the task to generate a parallel multi-level buffer operation based on the context.According to a further non-limiting embodiment, a method for processing data streams in a vehicle domain includes buffering data provided from a plurality of raw data streams output from a plurality of vehicles using a first level buffer installed on a first data processing level included in a system for parallel buffering and preprocessing data streams based on a multi-level context. The method further includes providing a serial data stream output from the first level buffer to an electronic task queue directory (TQD) module installed at a second data processing level included in the system for parallel buffering and preprocessing data streams based on a multi-level context. The method further includes creating a plurality of tasks via the TQD module in response to receiving the serial data stream. The TQD module is further configured to assign each task to a corresponding second-level buffer to separate the serial data stream into individual data values provided to a specific second-level buffer based on the task.In yet another non-limiting embodiment, a system for parallel buffering and preprocessing data streams based on multi-level context in vehicle domains includes a first-level buffer installed on a first data processing level. The first level buffer is configured to simultaneously rank data values provided by a plurality of data streams output from a plurality of vehicles. An electronic task queue directory (TQD) module is installed at a second data processing level different from the first data processing level, and the TQD module is configured to receive a serial data stream output from the first level buffer and separate the data values from the serial data stream based on a context of the data values. Second level buffers are installed at the second data processing level. The second level buffers have context buffer characteristics assigned thereto and are configured to rank data values having a context that matches the context buffer characteristic.In yet another non-limiting embodiment, a method for performing parallel buffering and preprocessing of data streams based on context in a vehicle domain includes simultaneously queuing data values provided from a plurality of data streams output by a plurality of vehicles at a first data processing level. The method further includes receiving a serial data stream at a second data processing level from a first level buffer and separating the data values from the serial data stream based on a context of the data values. The method further includes assigning a context buffer characteristic to each second level buffer and queuing data into respective second level data buffers based on a comparison between the context of the data values and the context buffer characteristics.In a further non-limiting embodiment, a computer program product controls a system for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains. The computer program product includes a computer readable storage medium having program instructions embodied therein and is executable by an electronic computer processor such that the system performs operations including buffering data provided from a plurality of raw data streams output by a plurality of vehicles using a first level buffer installed on a first data processing level included in a system for parallel buffering and preprocessing data streams based on a multi-level context. The operations further include providing a serial data stream output from the first level buffer to an electronic task queue directory (TQD) module installed at a second data processing level included in the system for parallel buffering and preprocessing data streams based on a multi-level context. The operation further includes creating, via the TQD module, a plurality of tasks in response to receiving the serial data stream. The operation further includes instructing the TQD module to assign each task to a corresponding second-level buffer to separate the serial data stream into individual data values provided to a specific second-level buffer based on the task.Additional functions are implemented by the techniques of the present invention. Other embodiments are described in detail herein and are considered part of the claimed invention. For a better understanding of the invention with the functions, reference is made to the description and to the drawings.BRIEF DESCRIPTION OF THE DRAWINGSFIG. 1 illustrates a cloud computing environment capable of supporting basic logic included in a system for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains, according to a non-limiting embodiment; FIG. 2 is a schematic illustration of a cloud computing node included in a distributed cloud environment; FIG. 3 is a set of functional abstraction layers provided by a cloud computing environment capable of supporting basic logic included in a system for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains, according to a non-limiting embodiment; FIG. 4 is a diagram illustrating operation of a system for parallel buffering and preprocessing of data streams based on multi-level context in vehicle domains according to a non-limiting embodiment; FIG. 5 is a diagram illustrating a system for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains according to a non-limiting embodiment; and FIG. 6 is a flow chart illustrating a method of controlling a system 100 for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains, according to a non-limiting embodiment.DETAILED DESCRIPTIONVarious embodiments provide a system for parallel buffering and preprocessing of datastreams based on a multi-level context capable of supporting preprocessing of datastreams based on context output by a plurality of vehicles, particularly during peak periods of data. The system for parallel buffering and preprocessing of data streams based on a multi-level context provides access to fast and large shared memory based on a data context (e.g., vehicle ID data, task ID data, timestamp data, etc.) for a single vehicleIn at least one embodiment, system performs parallel data processing between vehicles but serial processing for data output from each vehicle. Accordingly, at least one embodiment provides a system that saves about eight to about sixteen times memory usage on a multi-core computer (e.g., an 8-core computer) as compared to conventional process-level computing frameworks such as Hadoop, Store, and Spark. Parallel preprocessing buffering of data also eliminates concerns about low throughput caused by single ended CPU core usage to achieve high throughput with single ended CPU core usage.According to at least one embodiment, system for parallel buffering and preprocessing of data streams based on multi-level context may reduce latency in real-time site-related services during periods of peak data transfer rates by about 20% to about 30%. For example, during periods of peak data transfer rates, the system for parallel buffering and preprocessing of data streams based on a multi-level context may first pool large volume data in a first-level buffer queue and then quickly allocate buffered first-level data to a second-level buffer queue to perform preprocessing of data based on the context. In this way, the location-based services may discard useless data timely, thereby improving the performance and capacity of the computer system, server, and / or cloud network.Referring to FIG. 1, a cloud computing environment 10 is illustrated, according to a non-limiting embodiment, capable of supporting basic logic included in a system 100 for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains. As shown, cloud computing environment 10 includes one or more cloud computing nodes 50 with which local computing devices used by cloud customers, such as a personal digital assistant (PDA) or cellular telephone 54A, desktop computer 54B, laptop computer 54C, and / or motor vehicle computer system 54N, may communicate. In at least one embodiment, vehicle computing system 54N may include a mobile data terminal configured to exchange data with a cloud computing environment. The nodes 50 may communicate with each other. They may be physically or virtually grouped together (not shown) in one or more networks, for example, private, community, public, or hybrid clouds as described above, or a combination thereof. This enables cloud computing environment 10 to offer infrastructure, platforms, and / or software as services for which a cloud customer does not need to keep resources on a local computing device. It should be understood that the types of computing devices 54A-N shown in FIG. 2 are intended for purposes of illustration only and that computing nodes 50 and cloud computing environment 10 may communicate with any type of computerized device via any type of network and / or a network addressable connection (e.g., using a web browser).Referring to FIG. 2, a schematic diagram of a cloud computing node 50 included in a distributed cloud environment or cloud service network is shown, according to a non-limiting embodiment. Cloud computing node 50 is merely an example of a suitable cloud computing node and is not intended to limit the scope of use or functionality of embodiments of the invention described herein. Nevertheless, cloud computing node 50 may be implemented and / or perform all of the functionalities set forth hereinabove.Located within cloud computing node 50 is a computer system / server 12 that can operate with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with the computing system / server 12 include, but are not limited to, personal computing systems, server computing systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.The computer system / server 12 may be described in the general context of instructions executable by a computer system, such as program modules executed by a computer system. In general, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform particular tasks or implement particular abstract data types. The computer system / server 12 may be used in distributed cloud computing environments where tasks are performed by remote processing units interconnected by a communication network. In a distributed cloud computing environment, program modules may reside in both local and remote computer system storage media, including storage devices.As shown in FIG. 2, the computer system / server 12 is shown in cloud computing node 50 as a general purpose computing device. The components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components, such as system memory 28, to processor 16.Bus 18 represents one or more of any number of types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. For example, and without limitation, such architectures include an industry standard architecture (ISA) bus, a micro channel architecture (MCA) bus, an enhanced ISA (EISA) bus, a video electronics standards local association (VESA) bus, and a peripheral component interconnect (PCI) bus.Computer system / server 12 typically includes a variety of computer system readable media. Such media may be any available media that can be accessed by computer system / server 12 and includes both volatile and non-volatile media, removable and non-removable media.System memory 28 may include computer system readable media in the form of volatile memory such as random access memory (RAM) 30 and / or cache 32. Computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. For example only, a storage system 34 may be provided for reading from and writing to a non-replaceable, non-volatile magnetic medium (not shown and commonly referred to as a "hard disk"). Although not shown, a magnetic disk drive for reading from and writing to a replaceable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from and writing to a replaceable, non-volatile optical disk such as a CD-ROM, DVD-ROM, or other optical media, may be provided. In such cases, each may be connected to bus 18 by one or more data media interfaces. As will be further illustrated and described below, the memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the invention.A program / utility 40 having a set (at least one) of program modules 42 may be stored in memory 28, as an example, not by way of limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, the one or more application programs, the other program modules, and program data, and some combinations thereof, may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of embodiments of the invention as described herein.The computer system / server 12 may also be capable of interacting with one or more external devices 14 such as a keyboard, a pointing device, a display 24, etc., one or more devices that allow a user to interact with the computer system / server 12, and / or any devices (e.g., a network card, a modem, etc.) This allows computer system / server 12 to communicate with one or more other computing devices. Such data exchange may be performed via input / output (I / O) interfaces 22. Further, the computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or a public network (e.g., the Internet), via a network adapter 20. As shown, network adapter 20 communicates with the other components of computer system / server 12 via bus 18. It should be understood, although not shown, that other hardware and / or software components could be used with computer system / server 12. Examples include, but are not limited to: microcode, device drivers, redundant processing devices, external hard disk drive arrays, RAID systems, tape drives, and data archive storage systems.Referring to FIG. 3, a set of functional abstraction layers provided by the cloud computing environment 10 is shown. It should be understood from the outset that the components, layers, and functions shown in FIG. 3 are intended to be illustrative only and embodiments of the invention are not limited thereto. As depicted, the various layers and corresponding functions provided by the cloud computing environment 10 are described in more detail below.A hardware and software layer 60 contains hardware and software components. Examples of hardware components include mainframe computers, in one example zSeries® systems from IBM® ; reduced instruction set computer (RISC) architecture based servers, in one example IBM pSeries® systems; IBM xSeries® systems; IBM BladeCenter® systems; storage units, networks, and network components. Examples of software components include network application server software, in one example IBM's application server software WebSphere®; and database software, in one example IBM's database software DB2®. (IBM, zSerie, pSerie, xSerie, BladeCenter, WebSphere, and DB2 are trademarks of International Business Machines Corporation registered worldwide in many countries.)Virtualization layer 62 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers; virtual storage; virtual networks, including virtual private networks; virtual applications and operating systems; and virtual clients.In one example, management layer 64 may provide the functions described below. Resource provisioning provides for the dynamic creation of computing resources as well as other resources used to perform tasks within the cloud computing environment. Metering and pricing provide cost tracking when using resources within the cloud computing environment as well as billing for consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud users and tasks, as well as protection for data and other resources. A user portal provides users and system administrators with access to the cloud computing environment. Management of the service scope provides for the allocation and management of cloud computing resources so that the required service objectives are achieved. Planning and satisfying Service Level Agreement (SLA) provides the arrangement in advance and the acquisition of cloud computing resources for which a future request is anticipated according to an SLA.Workload layer 66 provides examples of the functionality for which the cloud computing environment may be used. Examples of workloads and functions that may be provided by this layer include: mapping and navigation; software development and lifecycle management; virtual classroom training; data analytics processing; and transaction processing.Although a cloud environment capable of supporting the basic logic of the system 100 for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains is described in detail above, it should be appreciated that the basic logic of the system 100 for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains may be locally located on one or more of the devices 54A- 54N. For example, the basic logic of the parallel buffering and preprocessing data streams system 100 based on a multi-level context may be installed in vehicle domains on each vehicle computer system 54N. In this manner, units 54A- 54N may locally perform the various functions and operations of system 100 for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains.Referring to FIG. 4, an operation of a system 100 for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains is illustrated according to a non-limiting embodiment. One or more vehicles 101 ato 101 dare in signal communication with the system 100 and output a respective real-time data stream 102 ato 102 d. According to a non-limiting embodiment, each data stream 101a-101d is output serially and includes a sequence of data values, such as tuple 103a-103d, that indicate the context of the data. The tuple 103 ato 103 dmay include, for example, <an ID of the vehicle 101 a, a task header ID, a travel ID of the vehicle 101 a, a time stamp of the vehicle 101 a, a position of the vehicle 101 a, a speed of the vehicle 101 a, a travel direction of the vehicle 101 a,... >, <an ID of the vehicle 101 b, a task header ID, a travel ID of the vehicle 101 b, a time stamp of the vehicle 101 b, a position of the vehicle 101 b, a speed of the vehicle 101 b, a travel direction of the vehicle 101 b,... >, etc. Each tuple 103 ato 103 din the data stream 102 ato 102 dof a respective vehicle 101 ato 101 dhas a sequence identifier such as a time stamp, for example. For example, the time stamp identifier is indicated by the numbers corresponding to each tuple, e.g., 1, 2, 3... 8, etc.The system 100 includes a first computing layer 104 supporting a first layer buffer 106 and a second computing layer 108 supporting one or more second layer buffers 110. In at least one embodiment, first computing layer 104 and second computing layer 108 are installed on a common server or cloud computing environment. The first level buffer 106 includes an input in signal communication with the vehicles 101 ato 101 dand an output in signal communication with the second level buffers 110 ato 110 d. In at least one embodiment, first level buffer 106 is configured as a serial buffer that simultaneously buffers data values provided by data streams 102a-102d. Accordingly, the first level buffer 106 may process data according to a first-in-first-out scheme.The second data processing layer 108 includes an electronic task, queue, and directory (TQD) module (not shown in FIG. 4 ) configured to control an operation of the second layer buffers 110 ato 110 d. Accordingly, the second data processing layer 108 determines whether a vehicle 101 ato 101 dhas requested a new task or an updated task based on the task header ID of a tuple 103 ato 103 d. When a new task is requested, the new task is allocated to a single second-level buffer 108. For example, a buffer 108 aof the second level may be assigned a buffer ID corresponding to the vehicle ID extracted from a tuple 103 ato 103 d. The queue length of the second-level buffers 110 ato 110 dmay be dynamically adjusted based on the throughput of the server or the network. When an updated task is requested, the previously allocated second level buffer is updated. In at least one embodiment, second computing layer 108 enables buffering of only a single task for a respective vehicle 101 ato 101 d. Accordingly, each vehicle 101 ato 101 dis assigned only a single second-level buffer 110 ato 110 duntil the current work task is completed.After a task has been assigned to a respective second level buffer 110a-110d, each tuple 103a-103d received at the second data processing level 108 is provided to the correct second level buffer 110a-110d. For example, tuple 103 aincluding a vehicle ID for a first vehicle 101 ais queued in a first buffer 110 ahaving a buffer ID matching the vehicle ID of first vehicle 101 a. In this way, in each buffer 110 ato 110 dof the second level, tuple 103 ato 103 dare queued, which are transmitted in the data stream in real time by a respective vehicle 101 ato 101 d. In other words, the data streams 102a-102d are separated according to the context of each tuple 103a-103d (e.g., vehicle ID) and are queued in respective second level buffers 110a-110d to establish a parallel buffering process based on the second level context. Thus, the non-limiting embodiment illustrated in FIG. 4 provides a second level first buffer 110 athat is queued in tuple 103 athat is transmitted by a first vehicle 101 ain the data stream, another second level buffer 110 bthat is queued in tuple 103 bthat is transmitted by a second vehicle 101 bin the data stream, a second level third buffer 110 cthat is queued in tuple 103 cthat is transmitted by a third vehicle 101 cin the data stream, and a second level fourth buffer 110 dthat is queued in tuple 103 dthat is transmitted by a fourth vehicle 101 din the data stream. Although four second level buffers 110 ato 110 dare illustrated, the invention is not limited thereto. The second data processing layer 108 may also load the tuple 103 ato 103 daccording to various priority policies such as a first-in-first-out scheme or based on the timestamp IDs included in the tuple 103 ato 103 d.The output of each second level buffer 110a-110d is in signal communication with a respective electronic message handler module 112a-112d. The message handler modules 112 ato 112 dreceed the queued tuple 103 ato 103 dand process the task of a particular vehicle 101 ato 101 dbased on the integrated task ID and the integrated vehicle ID. In at least one embodiment, message handler modules 112a-112d perform various processes, including, but not limited to, changing priority with which task is processed. For example, the processing priority of a task may be changed when the processing of data of each vehicle 101 ato 101 dhas been started. Once a message handler module 112a-112d completes a task corresponding to a respective one, the message handler module 112a-112d outputs a task completion signal to the second level buffer 110a-110d, and a new task corresponding to the respective vehicle is loaded into the corresponding second level buffer 110a-110d. In this way, the second computing layer 108 may only include a waiting task for each vehicle 101 ato 101 d. Accordingly, real-time data streams serially output from one or more vehicles 101 ato 101 dmay be preprocessed in parallel using the second-level buffers 110 ato 110 d. Moreover, the parallel buffering of the data stream is performed according to the vehicle domain specific context of the data stream.Referring to FIG. 5, a system 100 for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains is illustrated, according to a non-limiting embodiment. The system 100 includes a first level buffer 106, one or more electronic data porting modules 107a-107n, one or more second level buffers 110a-110n, a TQD electronic module 111, and one or more electronic message handler modules 112a-112n. The system 100 is in signal communication with one or more vehicles 101 ato 101 nvia a receiver 114. The receiver 114 and the first level buffer 106 are installed at a first data processing level, whereas the TQD module 111 and the second level buffers 110 ato 110 nare installed at a second data processing level different from the first data processing level. In at least one embodiment, data (e.g., tuple) from all vehicles 101 ato 101 dthat exchange data with network is queued in first level buffer 106, whereas data (e.g., tuple) output from a particular vehicle 110 ato 110 nare queued in second level buffers 110 ato 110 n.The system 100 begins an operation (i.e., an operation 1) by receiving real-time data streams from the vehicles 101 ato 101 n. Any means for exchanging data between the vehicles 101 ato 101 nand the receiver 114 may be used, including, but not limited to, data transmission via satellites. The receiver 114 provides data to the first level buffer serially, where they are queued simultaneously or sequentially (i.e., operation 2). As described above, the data streams generated by each vehicle include a sequence of individual data values, such as tuple, indicating, for example, content of a respective data stream. The context indicated by tuple may include, for example, an ID of a vehicle 101 ato 101 n, a task header ID, a travel ID of a vehicle 101 ato 101 n, a time stamp of a vehicle 101 ato 101 n, a position of a vehicle 101 ato 101 n, a speed of a vehicle 101 ato 101 n, and a travel direction of a vehicle 101 ato 101 n. In at least one embodiment, first level buffer is configured as a serial buffer for queuing raw data provided by receiver 114 according to a first-in-first-out scheme.Next, the buffered raw data is output from the first level buffer 106 to the data porting modules 107 ato 107 n(i.e., operation 3). In at least one embodiment, data porting modules 107 are installed on a core processing unit (CPU) and operate in parallel with each other. Each data porting module 107 ato 107 nis configured to obtain header messages (e.g., a task header ID) that are integrated into the raw data output from the first level buffer 106. Based on the header message, the data porting modules 107a-107n determine whether a new task should be created at the second data processing level or whether an existing task should be updated at the second data processing level. In at least one embodiment, data porting modules 107 ato 107 nmay determine that second data processing level currently includes a task associated with a particular vehicle and decide not to create another new task requested by vehicle until the existing task has been processed. In this way, the second computing level avoids creating multiple new tasks for each vehicle 101 ato 101 nuntil the existing task is completed.The TQD module 111 receives command signals from the data porting modules 107 ato 107 nthat instruct whether to create a new task at the second data processing level or whether to update an existing task currently being processed at the second data processing level (i.e., operation 4). For example, if a new task is requested, the new task is allocated to a single second-level buffer 110 a / 110 n(i.e., operation 5 a). Each second level buffer 110 ato 110 nmay be assigned a buffer ID corresponding to the vehicle ID extracted from a tuple of the data stream.The TQD module 111 may further perform various management operations that control an operation of the second-level buffers 110 ato 110 n. For example, the TQD module 111 may load the tuple into its respective second level buffers 110 ato 110 naccording to various priority policies such as a first-in-first-out scheme or based on the timestamp IDs. The TQD module 111 may further dynamically adjust the queue length of the second-level buffers 110 ato 110 dbased on the throughput of the server or the network.When an updated task is requested, the previously allocated second level buffer 110 bis updated (i.e., operation 5 b). In at least one embodiment, TQD module 111 enables buffering of only a single task for a respective vehicle 101 ato 101 d, as mentioned above. Accordingly, each vehicle 101 ato 101 nis assigned only a single second-level buffer 110 ato 110 nuntil the previously created task currently being processed in the second-level buffer 110 ato 110 nis completed.The second level buffers 110 ato 110 noutput the queued tuple to define a separate data stream indicating a task requested by a respective vehicle 101 ato 101 d. Each separate data stream is provided to an electronic work task directory (WTD) module 116 (i.e., operation 6). The WTD module 116 contains data or definitions about the structure of each separate data stream, thereby allowing for the unpacking of fields from the message content for verification. In at least one embodiment, WTD module 116 queries message header or field within separated data stream to retrieve subject classification or keyword. The item classification or keyword can then be used to forward the separated data stream to a corresponding message handler module 112a-112n (i.e., operation 7).The message handler modules 112a-112n are executable in response to the decisions of the WTD module 116. For example, the WTD module 116 analyzes the separated data streams and determines a specific message handler module 112 ato 112 nconfigured to perform processing operations capable of performing a task requested by a respective vehicle 101 ato 101 n.For example, all tasks that include image content or for which image editing of that content is required are provided to a message handler module 112a-112n that is configured to edit images. Similarly, all tasks that include audio data that requires audio editing are passed to a message handler module 112a-112n that is configured to edit audio data. Tasks that include other recognizable data types are passed to a respective message handler module 112a-112n that is capable of processing the task. This operation may involve providing a separate data stream through a sequence of processing steps, each performed by a different message handler module 112a-112n, if the message includes a series of content types. Tasks containing executable code may be passed to a message handler module 112a-112n, which performs authentication (e.g., digital signature verification or virus verification) to protect subscribers from unwanted program code prior to publication approval.In at least one embodiment, processing performed by a selected message handler module 112a-112n is determined by specified requests from individual subscriber applications, respective features of requested task, and content of a separate data stream, as well as subscriber system capabilities. For example, a task ID integrated into a tuple of the separate data stream may indicate the presence of specific multimedia content and reveal the features of the content that would be relevant to a decision on how to process a separate data stream. In some cases, the task ID could also indicate one or more code modules that would be required to adequately process the detected content. The selected message handler module 112a-112n then performs operations on the separated data stream in accordance with the associated task definition. According to at least one embodiment, the result, i.e. the completed task, may subsequently be returned to the corresponding vehicle 101 ato 101 nthat had handed over the requested task.Referring to FIG. 6, a method of controlling a system 100 for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains is illustrated, according to a non-limiting embodiment. The method begins at operation 600, and at operation 602, raw data streams output by one or more vehicles are received. In at least one embodiment, vehicles output data streams in series. At operation 604, the raw data streams are buffered at a first data processing level. According to at least one embodiment, raw data streams include tuple, and the tuple is serially buffered in a first level buffer according to a first-in-first-out scheme. At operation 606, a task is determined based on the data context derived from tuple output from the first level buffer. In at least one embodiment, each tuple includes a topic ID that indicates task. At operation 608, an analysis is performed to determine whether the determined task is currently present at a second data processing level of the system. If the task is present, the present task is updated at the second data processing level in operation 610. However, if the determined task is not present at the second data processing level, a new task is created in operation 612. In at least one embodiment, second data processing level processes only a single existing task for each vehicle that exchanges data with system until existing task being processed is completed.In operation 614, a second level buffer is allocated to the newly created task. In at least one embodiment, second level buffer is assigned a buffer ID that matches a vehicle ID corresponding to vehicle requesting task. In operation 616, a TQD module installed on the second data processing level performs parallel data buffering of tasks corresponding to the raw data streams provided by the first data processing level. In this way, data streams can be output serially by the vehicles, whereas preprocessing buffering of data according to context-based analysis is performed in parallel.In operation 618, an analysis is performed to determine whether the buffering process of one or more second level buffers is completed. If buffering of a task is still continuing, the method returns to operation 616 to continue buffering the data (e.g., the tuple) of the corresponding task. However, when the buffering is complete, a separate data stream is generated and provided to a WTD module in operation 620. In operation 622, the WTD module determines one or more task handler modules configured to process the separated data stream. Subsequently, the separated data stream is provided to the respective task handler modules for processing in operation 624. In operation 626, the results of the completed task are reported and the method ends in operation 628.According to various non-limiting embodiments described above, a system for parallel buffering and preprocessing of data streams based on a multi-level context is configured to first gather large volume data in a first-level buffer queue and then quickly allocate buffered first-level data to a second-level buffer queue to perform preprocessing of data based on the context. In this way, latency in real-time site-based services during periods of peak data transfer rates can be reduced by about 20% to about 30%. Accordingly, the performance and capacity of the computer system, server, and / or cloud network is significantly improved compared to conventional stream processing systems and / or existing internet-based frameworks.As used herein, a computer readable storage medium is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through an optical fiber cable), or electrical signals transmitted through a line.As used herein, "module" refers to an application specific integrated circuit (ASIC), an electronic circuit, a (shared, dedicated, or group) hardware electronic computer processor, and memory that executes one or more software or firmware programs, combinatorial circuitry, a microprocessor, and / or other suitable components that provide the described functionality. When implemented in software, a module may be embodied in memory as a non-transitory machine readable storage medium readable by processing circuitry and storing instructions for execution by the processing circuitry to perform a method.The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application, or the technical improvement over technologies available on the market, or to enable others skilled in the art to understand the embodiments disclosed herein.The present invention may be a system, a method and / or a computer program product. The computer program product may include computer readable storage medium(s) on / on which computer readable program instructions are / are stored for causing a processor to perform aspects of the present invention.The computer readable storage medium may be a physical device that can retain and store instructions for use by a system for executing instructions. The computer readable storage medium may be, for example, but is not limited to, an electronic storage unit, a magnetic storage unit, an optical storage unit, an electromagnetic storage unit, a semiconductor storage unit, or any suitable combination thereof. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions stored thereon, and any suitable combination thereof. A computer readable storage medium, as used herein, is not intended to be transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses conducted through a fiber optic cable), or electrical signals transmitted through a wire.Computer readable program instructions described herein may be downloaded from a computer readable storage medium to respective data processing / processing devices or via a network such as the Internet, a local area network, a wide area network, and / or a wireless network to an external computer or storage device. The network may include copper transmission cables, lightwave transmission conductors, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing unit receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the corresponding computing / processing unit.Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including object oriented programming languages such as Smalltalk, C++, or the like, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, via the Internet using an Internet Service Provider). In some embodiments, electronic circuits, including, for example, programmable logic circuits, field programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuits to perform aspects of the present invention.Aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It is noted that each block of the flowcharts and / or the block diagrams, and combinations of blocks in the flowcharts and / or the block diagrams, may be executed by computer readable program instructions.These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions executed via the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions / steps specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored on a computer readable storage medium that can control a computer, programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored thereon comprises an article of manufacture including instructions that implement aspects of the function / step specified in the flowchart and / or block diagrams block or blocks.The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of process steps to be performed on the computer or other programmable apparatus or other device to produce a computer executed process, such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / steps specified in the flowchart and / or block diagram block or blocks.The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions that include one or more executable instructions for executing the particular logical function(s). In some alternative implementations, the functions specified in the block may occur in a different order than shown in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order depending on the appropriate functionality. It is further noted that each block of the block diagrams and / or the flowcharts, as well as combinations of blocks in the block diagrams and / or the flowcharts, may be implemented by special purpose hardware-based systems that perform the specified functions or steps, or perform combinations of special purpose hardware and computer instructions.

Claims

A system (100) for parallel buffering and preprocessing data streams based on a multi-level context in vehicle domains, comprising: a first data processing level (104) including a first level buffer (106) configured to buffer data provided from a plurality of raw data streams (102a-102d) output from a plurality of vehicles (101a-101d); and a second data processing layer (108) including an electronic task queue directory (TQD) module and a plurality of second layer data processing buffers, the TQD module configured to generate a plurality of tasks in response to receiving a serial data stream output from the first layer buffer (106), wherein the TQD module assigns each task to a corresponding second layer buffer and separates the serial data stream into individual data values provided to a specific second layer buffer (108) based on the task.The system (100) of claim 1, wherein the TQD module provides the data values in parallel to the corresponding second level buffers.The system (100) of claim 2, wherein the data values include vehicle characteristics and the TQD module assigns a buffer characteristic to each second level buffer such that the data values are loaded into a corresponding second level buffer having a buffer characteristic that matches the vehicle characteristics.The system (100) of claim 2, wherein the TQD module limits each vehicle (101a-101d) to a single work task created at the second data processing level.The system (100) of claim 2, wherein the TQD module loads each second level buffer based on a time stamp indicated by a respective data value.A method (600) of processing data streams in a vehicle domain, the method (600) comprising: buffering data provided from a plurality of raw data streams output from a plurality of vehicles (101a-101d) using a first level buffer (108) installed on a first data processing level (104) included in a system for parallel buffering and preprocessing of data streams based on a multi-level context; providing a serial data stream output from the first level buffer (106) to an electronic task queue directory (TQD) module installed on a second data processing level (108) included in the system (100) for parallel buffering and preprocessing data streams based on a multi-level context; creating a plurality of tasks via the TQD module in response to receiving the serial data stream, wherein the TQD module assigns each task to a corresponding second level buffer (110a-110d) to separate the serial data stream into individual data values provided to a specific second level buffer based on the task.The method of claim 6, wherein the TQD module provides the data values to the corresponding second level buffers (110a to 110d) in parallel.The method of claim 7, further comprising integrating vehicle characteristics into the data values, wherein the TQD module assigns a buffer characteristic to each second level buffer (110a-110d) such that the data values are loaded into a corresponding second level buffer (110a-110d) having a buffer characteristic that matches the vehicle characteristics.The method of claim 7, wherein the TQD module limits each vehicle (101a-101d) to a single work item created at the second computing level (108).The method of claim 7, wherein the TQD module loads each second level buffer (101a to 101d) based on a time stamp indicated by a respective data value.A system for parallel buffering and preprocessing data streams based on a multi-level context in vehicle domains, comprising: a first-level buffer (106) installed on a first data processing level (104), wherein the first-level buffer (106) is configured to simultaneously rank data values provided from a plurality of data streams (101a-101d) output from a plurality of vehicles; an electronic task queue directory (TQD) module installed on a second data processing level (108) different from the first data processing level (104), the TQD module configured to receive a serial data stream output from the first level buffer (106) and separate the data values from the serial data stream based on a context of the data values; and a plurality of second level buffers installed on the second data processing level, the second level buffers having context buffer characteristics assigned thereto and configured to rank data values having a context matching the context buffer characteristic.The system of claim 11, wherein the TQD module loads data values into respective second level buffers according to a scheme for parallel providing data to generate respective separate data streams.The system of claim 12, further comprising at least one electronic data porting module configured to determine a task requested by a respective vehicle based on the context of the data values, and to direct the TQD module to perform a creation of a new task at the second data processing level (108) and / or an update of an existing task in response to determining the requested task.The system of claim 13, further comprising: at least one electronic message handler configured to execute at least one data process required to complete a requested task; and an electronic work task directory (WTD) module configured to receive a separate data stream, determine at least one process required to complete the requested task corresponding to the separate data stream, and provide the separate data stream to the at least one message handler based on the at least one determined process.The system of claim 14, wherein the TQD module assigns a single work task for each vehicle (101a-101d) to a single second level buffer and updates the second level buffer (108) to include a new task for a respective vehicle (101a-101d) after the work task is completed.A method for performing parallel buffering and preprocessing of data streams (102a to 102d) based on context in a vehicle domain, the method comprising: simultaneously queuing data values provided from a plurality of data streams (102a to 102d) output from a plurality of vehicles (101a to 101d) at a first data processing level (104); receiving a serial data stream at a second data processing level (108) from a first level buffer (106) and separating the data values from the serial data stream based on context of the data values; and assigning a context buffer characteristic value to each second level buffer; and queuing data into respective second level data buffers based on a comparison between the context of the data values and the context buffer characteristics.The method of claim 16, further comprising loading data values according to a scheme for parallel providing data into respective second level buffers to generate respective separate data streams.The method of claim 17, further comprising determining a task requested by a respective vehicle based on the context of the data values and instructing an electronic task queue directory (TQD) module to perform a creation of a new task at the second data processing level and / or an update of an existing task in response to the determination of the requested task.The method of claim 18, further comprising: determining at least one process required to complete the requested task corresponding to the separated data stream; and providing the separated data stream to the at least one message handler module based on the at least one determined process such that at least one data process required to complete a requested task is executed.The method of claim 19, further comprising: assigning a single second level buffer to a single work task for each vehicle (101a-101d); and updating the second level buffer to include a new task for a respective vehicle (101a-101d) after the work task is completed.A computer program product for controlling a system for parallel buffering and preprocessing of data streams based on a multi-level context in vehicle domains, the computer program product comprising a computer readable storage medium having program instructions embodied therein, the program instructions executable by an electronic computer processor such that the system performs operations comprising: buffering (604) data provided from a plurality of raw data streams (102a to 102d) output from a plurality of vehicles (101a to 101d) using a first level buffer (106) installed on a first data processing level (104) included in a system for parallel buffering and preprocessing of data streams (102a to 102d) based on a multi-level context; providing a serial data stream output from the first level buffer to an electronic task queue directory (TQD) module installed on a second data processing level (108) included in the system for parallel buffering and preprocessing data streams based on a multi-level context; creating a plurality of tasks via the TQD module in response to receiving the serial data stream; and instructing the TQD module to assign each task to a corresponding second level buffer to separate the serial data stream into individual data values provided to a specific second level buffer based on the task.The computer program product of claim 21, wherein the operations further comprise providing the data values to the respective second level buffers in parallel.The computer program product of claim 22, wherein the operations further comprise integrating vehicle characteristics into the data values and assigning a buffer characteristic to each second level buffer such that the data values are loaded into a corresponding second level buffer having a buffer characteristic that matches the vehicle characteristics.The computer program product of claim 22, wherein the operations further comprise limiting each vehicle (101a-101d) to a single work task created at the second data processing level (108).The computer program product of claim 22, wherein the operations further comprise loading each second level buffer based on a time stamp indicated by a respective data value.

Images