DATA PROCESSING SYSTEM, DATA PROCESSING METHOD, PROCESSING DEVICE, AND PROGRAM

Abstract

To accelerate response in a lambda architecture.SOLUTION: A data processing system which stream processes received data, and batch processes the data after having accumulated, includes: a high-speed response processing device 11 for executing a part of a stream process with respect to the received data; a high-speed response processing result providing device 12 for retaining a processing result of the high-speed response processing device 11; a real time processing device 21 for executing a remaining process of the stream process with respect to the processing result of the high-speed response processing device 11; a storage 31 for accumulating received data; and a batch processing device 32 for batch processing the data accumulated in the storage 31.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a data processing system, a data processing method, a processing device, and a program. [Background technology]

[0002] In recent years, with the spread of the Internet of Things (IoT), various devices in our daily lives are now connected to the Internet. By connecting applications deployed on cloud servers to IoT devices, even if the computing power of the IoT device is low, the entire system can perform complex processing such as analyzing data and changing operations according to the analysis results.

[0003] As the number of IoT terminals increases, the amount of data sent to cloud servers is also increasing, causing problems such as rising network costs and increased load on applications on cloud servers. In addition, connected cars and IoT-enabled factory robots require faster responses from servers. When the server and terminal are physically far apart, it is difficult to speed up the response.

[0004] Against this background, edge computing, in which servers are deployed at the periphery (edge) of a network and parts of applications are placed on these servers, is attracting attention. Using an edge computing environment can reduce the amount of data sent to cloud servers and speed up responses to terminals. For example, by placing the processing group (hereinafter referred to as logic) of an application that aggregates information sent by IoT terminals on an edge server, the amount of communication required to send data to a cloud server can be reduced. By placing logic that frequently communicates with terminals on an edge server, the response time of the application can be shortened.

[0005] In order to run an application in a distributed manner on multiple machines, it is necessary to divide the application into multiple pieces of logic and place them on each machine (see Non-Patent Document 1). However, dividing and placing an application requires a design that takes into account the application's operating environment and the processing content of the logic, which places a heavy burden on developers.

[0006] On the other hand, a model called lambda architecture is known as a configuration of an application that processes stream data, including IoT data, in real time (see Non-Patent Documents 2 and 3). In the lambda architecture, it is possible to obtain both detailed and immediate processing results by simultaneously performing batch processing and stream processing on a very large data set. The lambda architecture is used in several commercial applications for IoT. By implementing an application based on the lambda architecture, a user or terminal that queries the processing results can obtain temporary processing results in real time, and can also obtain highly accurate and detailed processing results as soon as batch processing is completed. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Alex Reznik, et al., “Developing Software for Multi-Access Edge Computing,” ETSI White Paper, No. 20, Sept. 2017. [Non-Patent Document 2] Nathan Marz, “Dig Data Lambda Architecture,” Internet〈 URL:http: / / www.databasetube.com / database / big-data-lambda-architecture / 〉 [Non-Patent Document 3] Dutta, K. and Jayapal, M., “Big data analytics for real time systems,” Big data analytics seminar, pp.1-13, 2015. Summary of the Invention [Problem to be solved by the invention]

[0008] When creating an application based on the lambda architecture in an edge computing environment, the response time can be shortened by simply executing stream processing on the edge server. However, the lambda architecture does not assume further acceleration of stream processing.

[0009] If the user or terminal that uses the processing results can tolerate a decrease in the accuracy of the processing results, it is possible to expect even faster responses. However, since the lambda architecture does not anticipate further speed-ups in stream processing, a mechanism that can achieve this is required.

[0010] The present invention has been made in view of the above, and has an object to increase the response speed in a lambda architecture. [Means for solving the problem]

[0011] A data processing system according to one aspect of the present invention is a data processing system that performs stream processing on received data and batch processes the received data after storing the data, the data processing system including: a first processing device that performs a first processing step of the stream processing on the data and stores a processing result; a second processing device that performs a second processing step of the stream processing on the processing result of the first processing device; a storage device that stores the received data; and a batch processing device that performs batch processing on the data stored in the storage. The first processing step is a processing group extracted from the stream processing in order of the earliest processing step, and the second processing step is a processing group extracted from the stream processing after the first processing step. . Effect of the Invention

[0012] According to the present invention, it is possible to achieve high speed response in a lambda architecture. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing an overview of a data processing system according to this embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of a quick response processing device and a quick response processing result providing device. [Diagram 3] FIG. 3 is a diagram for explaining an embodiment in which an obstacle is detected based on an image from an on-board camera and CAN data. [Figure 4] FIG. 4 is a diagram illustrating an example of the arrangement of each layer in an edge computing environment. [Diagram 5] FIG. 5 is a diagram illustrating another example of the arrangement of layers in an edge computing environment. [Figure 6] FIG. 6 is a diagram illustrating an example of a hardware configuration of the quick response processing device and the quick response processing result providing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] The data processing system of this embodiment will be described with reference to Fig. 1. The data processing system shown in the figure is composed of a high-speed response processing layer, a real-time processing layer, a batch processing layer, and a processing result providing layer. In the data processing system of this embodiment, a new high-speed response processing layer has been added so that a part of the processing result can be referred to before the processing result of the real-time processing layer is output.

[0016] Data to be processed is sent from each of a large number of terminals such as sensors. The data to be processed is sent to both the batch processing device and the high-speed response processing layer. The batch processing layer accumulates the received data and processes the data in batches at specified intervals. The high-speed response processing layer and real-time processing layer stream process the received data.

[0017] In this embodiment, a part of the stream processing executed by the real-time processing layer is divided as processing in the high-speed response processing layer. Specifically, based on the data flow in the real-time processing layer, the processing is extracted as processing in the high-speed response processing layer in order of the earliest processing order. The above procedure is repeated to determine the processing in the high-speed response processing layer so that the total processing time of the high-speed response processing layer does not exceed a predetermined threshold. In the high-speed response processing layer, the processing divided from the real-time processing layer is executed, and the processing result is sent to the real-time processing layer and is held so that it can be returned to an inquiry from a client. The real-time processing layer takes over the data and the processing result, and executes the remaining processing of the stream processing.

[0018] The fast response processing layer includes a fast response processing device 11 and a fast response processing result providing device 12. The real time processing layer includes a real time processing device 21 and a real time processing result providing device 22. The batch processing layer includes a storage 31 and a batch processing device 32. The processing result providing layer includes a batch processing result providing device 33. Each device provided in the data processing system will be described below.

[0019] The quick response processing device 11 receives data and processes the data in front of the real-time processing device 21. The quick response processing device 11 transmits a processing result (hereinafter, referred to as a quick response processing result) to the real-time processing device 21 and the quick response processing result providing device 12.

[0020] The quick response processing result providing device 12 holds the quick response processing result received from the quick response processing device 11, and returns the quick response processing result in response to an inquiry from a client.

[0021] The real-time processing device 21 processes data based on the high-speed response processing result, and transmits the processing result (hereinafter referred to as real-time processing result) to the real-time processing result providing device 22. The real-time processing device 21 may transmit the real-time processing result to the high-speed response processing result providing device 12. Upon receiving the real-time processing result, the high-speed response processing result providing device 12 modifies the high-speed response processing result based on the real-time processing result.

[0022] The real-time processing result providing device 22 holds the real-time processing results and returns the real-time processing results in response to an inquiry from a client.

[0023] The storage 31 holds all data to be processed. When the batch processing layer receives data, the data is stored in the storage 31.

[0024] The batch processing device 32 processes all the data held in the storage 31 at a predetermined timing, and transmits the processing results (hereinafter referred to as batch processing results) to a batch processing result providing device 33.

[0025] The batch processing result providing device 33 holds the batch processing results and returns the batch processing results in response to an inquiry from a client.

[0026] The quick response processing device 11 and the quick response processing result providing device 12 included in the quick response processing layer will be described with reference to Fig. 2. The quick response processing device 11 shown in Fig. 2 includes a receiving unit 111, a processing unit 112, and a transmitting unit 113. The quick response processing result providing device 12 includes a correcting unit 121, a storage unit 122, and a responding unit 123.

[0027] The receiver 111 receives the data, which is also sent to the batch processing layer.

[0028] The processing unit 112 processes the received data to obtain a high-speed response processing result. This processing is part of the processing from data to obtaining a real-time processing result, and is completed within a predetermined time.

[0029] The transmission unit 113 transmits the high-speed response processing result to the real-time processing layer and the high-speed response processing result providing device 12 .

[0030] The storage unit 122 stores the quick response processing result received from the quick response processing device 11 .

[0031] The response unit 123 returns the high-speed response processing result stored in the storage unit 122 in response to an inquiry from a client.

[0032] The correcting unit 121 receives real-time processing results from the real-time processing layer and batch processing results from the batch processing layer, and corrects the high-speed response processing results stored in the storage unit 122. For example, when the correcting unit 121 receives real-time processing results from the real-time processing layer, it deletes the high-speed response processing results corresponding to the real-time processing results from the storage unit 122. Alternatively, the correcting unit 121 may discard the high-speed response processing results and hold the real-time processing results.

[0033] With reference to FIG. 3, an example in which an obstacle is detected based on an on-board camera image and Control Area Network (CAN) data transmitted by a vehicle using the data processing system of this embodiment will be described.

[0034] When a conventional lambda architecture system is used, the batch processing layer accumulates data received from vehicles and uses data from multiple vehicles to determine the type and position of an obstacle. When data is received from a vehicle, the real-time processing layer performs object detection and recognition processing and position estimation processing to determine the type and position of an obstacle. This example is applied to the data processing system of this embodiment. In the example of FIG. 3, object detection and recognition processing from vehicle-mounted camera images is divided into a high-speed response processing layer.

[0035] The following describes the process of the embodiment shown in Fig. 3. Fig. 3 shows the data flow, the process executed in each layer, and the data held in each layer.

[0036] In step S11, the vehicle transmits the on-board camera image and the CAN data to the high-speed response processing layer, and in step S12, the vehicle transmits the on-board camera image and the CAN data to the batch processing layer.

[0037] In step S13, the high-speed response processing layer detects an object from the vehicle-mounted camera image and recognizes the object. For example, in this process, the presence or absence of an obstacle, the type of the obstacle, and the approximate position of the obstacle are obtained as the high-speed response processing result. The approximate position is, for example, the shooting position and shooting direction of the vehicle-mounted camera image. The vehicle-mounted camera image, CAN data, and the high-speed response processing result are transmitted to the real-time processing layer.

[0038] In step S14, the high-speed response processing result obtained in step S13 is accumulated in the high-speed response processing layer. The high-speed response processing layer becomes able to reply to an inquiry about the high-speed response processing result. For example, when a client (another vehicle) inquires of the high-speed response processing layer about whether or not an obstacle exists at a certain position, the high-speed response processing layer searches for a high-speed response processing result indicating that an obstacle exists at that position, and replies with the searched high-speed response processing result.

[0039] In step S15, the real-time processing layer estimates the exact position of the obstacle detected from the vehicle-mounted camera image based on the vehicle-mounted camera image, the CAN data, and the high-speed response processing result.

[0040] In step S16, real-time processing results including the type of obstacle obtained in step S13 and the position of the obstacle obtained in step S15 are accumulated, and the real-time processing layer is able to reply to an inquiry about the real-time processing results.

[0041] In step S17, the real-time processing result is sent to the high-speed response processing layer, which corrects the high-speed response processing result based on the real-time processing result.

[0042] On the other hand, in step S21, the batch processing layer holds the received data.

[0043] In step S22, the batch processing layer performs object detection / recognition processing and position estimation processing based on the vehicle-mounted camera images and CAN data from multiple vehicles when a predetermined interval or a predetermined number of data items have been accumulated.

[0044] In step S23, the batch processing results including the types and positions of the obstacles obtained in the batch processing layer are accumulated in the processing result providing layer, so that the processing result providing layer can reply to an inquiry about the batch processing results.

[0045] In step S24, the batch processing layer sends the rapid response processing result to the rapid response processing layer, which modifies the rapid response processing result based on the batch processing result.

[0046] Here, the time reduction when using the data processing system of this embodiment will be described. For example, in the real-time processing layer of the conventional lambda architecture, it takes 6 seconds to acquire the type and position of an obstacle. The breakdown of the processing time is as follows:

[0047] Transmission of vehicle camera images and CAN data to the cloud: 2 seconds Object detection and recognition processing from vehicle-mounted camera images: 1 second Detected object position estimation process: 3 seconds

[0048] In the example shown in Figure 3, object detection and recognition processing is divided into a high-speed response processing layer, and the results of object detection and recognition processing can be referenced. This reduces the time from when the vehicle transmits data to when information can be obtained from the high-speed response processing layer by three seconds.

[0049] In addition, by placing the high-speed response processing layer at the edge, if the time it takes to process the transmission of vehicle camera images and CAN data to the edge is 1 second, the time can be reduced by another 1 second.

[0050] Queries to the high-speed response processing layer can also be shortened. For example, assume that the time it takes for a client to query real-time processing results is 1 second, of which the communication delay to the cloud is 0.25 seconds and the information search is 0.5 seconds. If the communication delay to the edge is 0.1 seconds, the time it takes for a client to query high-speed response processing results can be shortened by 0.3 seconds.

[0051] The embodiment of FIG. 3 can be realized in an edge computing environment with a tree structure rooted in the cloud, as shown in FIG. 4. Vehicles are connected to a base station for communication and transmit data and queries. An edge server is deployed near the base station. A high-speed response processing layer is placed in the edge server, and a real-time processing layer, a batch processing layer, and a processing result provision layer are placed in the cloud.

[0052] The data transmitted by the vehicle is processed by an edge server near the base station to which the vehicle is connected, and is then sent to the cloud's batch processing layer. The results of the high-speed response processing obtained by the high-speed response processing layer are stored in the edge server so that they can be used to respond to inquiries, and are also sent to the cloud's real-time processing layer.

[0053] When a vehicle inquires about the results of the rapid response processing, the edge server of the base station to which the vehicle is connected returns the results of the rapid response processing. The results of the rapid response processing held by the edge servers of other base stations to which the vehicle is not connected can be obtained via the cloud.

[0054] When a vehicle wants to view real-time and batch processing results, it queries the cloud.

[0055] Furthermore, when a vehicle (terminal) has high computing power, a high-speed response processing layer may be placed in each vehicle, as shown in Fig. 5. In Fig. 5, the real-time processing layer is placed in an edge server, and the batch processing layer and the processing result providing layer are placed in the cloud.

[0056] Each vehicle performs high-speed processing on the data obtained from the sensors installed in the vehicle and sends it to the batch processing layer of the cloud. Each vehicle sends the results of the high-speed processing to the real-time processing layer and also stores them in the vehicle.

[0057] A vehicle can obtain a high-speed response processing result held by another vehicle via an edge server. When direct communication between vehicles is possible, a vehicle may obtain a high-speed response processing result directly from another vehicle.

[0058] As described above, the data processing system of this embodiment is a data processing system that applies a lambda architecture that performs stream processing on received data and batch processing after accumulating the data, and includes a high-speed response processing device 11 that executes part of the stream processing on the received data, and a high-speed response processing result providing device 12 that holds the processing result of the high-speed response processing device 11 and replies to an inquiry. This allows the client to refer to the processing result of the real-time processing layer, which has lower accuracy than batch processing but has a shorter processing time, until the processing result of the batch processing in the batch processing layer is output, and further allows the client to refer to the processing result of part of the stream processing before the processing result of the stream processing in the real-time processing layer can be referred to. Since it is possible to refer to the processing result faster than the conventional lambda architecture, if a decrease in accuracy of the processing result is acceptable, the response can be accelerated.

[0059] In the data processing system of this embodiment, stream processing is divided into processes in the high-speed response processing layer in order of the earliest processing order so that the total processing time does not exceed a predetermined threshold, making it possible to present a general-purpose division and placement of processing, and reducing the burden associated with the division and placement of applications.

[0060] The above-described quick response processing device 11 and quick response processing result providing device 12 may be implemented as a general-purpose computer system including a central processing unit (CPU) 901, a memory 902, a storage 903, a communication device 904, an input device 905, and an output device 906, as shown in Fig. 6. In this computer system, the quick response processing device 11 and the quick response processing result providing device 12 are realized by the CPU 901 executing a predetermined program loaded onto the memory 902. This program can be recorded on a computer-readable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or can be distributed via a network. The quick response processing device 11 and the quick response processing result providing device 12 may be implemented as a single device. [Explanation of symbols]

[0061] 11...High-speed response processing device 111…Receiver 112... Processing section 113...Transmitter 12...High-speed response processing result providing device 121…Revision section 122...Storage section 123...Response section 21...Real-time processing device 22...Real-time processing result providing device 31…Storage 32...Batch processing equipment 33...Batch processing result providing device

Claims

1. A data processing system that performs stream processing on received data and batch processes the received data after storing the data, a first processing device that executes a first processing step of the stream processing on the data and stores a processing result; a second processing device that executes a second processing step of the stream processing on a processing result of the first processing device; A storage device for storing the received data; a batch processing device for batch processing the data stored in the storage, The first processing step is a processing group extracted from the stream processing in order of the earliest processing step, and the second processing step is a processing group extracted from the stream processing after the first processing step. Data processing system.

2. 2. The data processing system of claim 1, a first result providing device that holds the processing result of the first processing device and responds to an inquiry; a second result providing device that holds the processing result of the second processing device and responds to the inquiry; a third result providing device that holds the processing results of the batch processing device and responds to inquiries. Data processing system.

3. A data processing method for performing stream processing on received data and batch processing on the received data after storing the data, comprising the steps of: By computer executing a first processing step of the stream processing on the data and storing a processing result; executing a second processing step of the stream processing on a processing result of the first processing step; storing the received data in a storage device; and performing batch processing on the data stored in the storage. The first processing step is a processing group extracted from the stream processing in order of the earliest processing step, and the second processing step is a processing group extracted from the stream processing after the first processing step. Data processing methods.

4. A processing device provided in a data processing system that performs stream processing on received data and batch processes the received data after storing the data, a processing unit that executes a part of the stream processing on the data; A storage unit for storing a processing result of the processing unit; A response unit that returns the processing result in response to an inquiry, The part of the processing steps of the stream processing is a processing group extracted from the stream processing based on the processing order and the total processing time. Processing unit.

5. 5. The processing device according to claim 4, a correction unit that corrects the processing result stored in the storage unit based on at least one of the processing result of the stream processing and the processing result of the batch processing; Processing unit.

6. A processing device provided in a data processing system that performs stream processing on received data and batch processes the received data after storing the data, a processing unit that receives from a first processing device a first processing result for the data of a processing group extracted in order from the earliest processing order of the stream processing, and executes a processing of the stream processing subsequent to the processing group on the first processing result; a storage unit for storing a second processing result of the stream processing; a response unit that returns the second processing result in response to an inquiry. Processing unit.

7. 7. A program for causing a computer to operate as the processing device according to claim 4.

Images