System, method, and program for collecting data

Abstract

The purpose of this disclosure is to provide a system that does not require a large amount of network resources even when collecting data from a large number of data sources.　This disclosure provides a system for collecting, in a server, data from multiple data sources, wherein the respective data sources transmit a data transfer requirement to a controller, the controller determines a transmission policy for each data source on the basis of the data transfer requirement, a data source autonomously transmits a communication path connection request on the basis of the transmission policy, the controller generates a communication path on the basis of the communication path connection request from the data source, and the data source transmits data using the generated communication path.

Description

Technical Field

[0001] The present disclosure relates to communication control for data collection.

Background Art

[0002] User terminals, sensors, and in-vehicle systems have been developing, and the number of sensors and the amount of data generated from each sensor have increased significantly. In the conventional communication method for collecting such data, the communication processing on the receiving side may become a bottleneck.

[0003] As a high-speed data transfer method, RDMA (Remote Direct Memory Access) has been considered. In RDMA, DMA transfer of data (direct data transfer between peripheral devices and main memory (RAM) etc. without going through the CPU) is performed from the memory of a local computer to the memory of a different remote computer. Therefore, since RDMA does not require CPU processing for data transfer, the bottleneck on the receiving side can be avoided.

[0004] A remote transfer technology using this RDMA has been proposed (see, for example, Non-Patent Document 1). However, in Non-Patent Document 1, in order to transfer data by RDMA in a wide-area communication network, it is necessary to set a communication path for each flow in advance. Therefore, when collecting data from a large number of data sources, a large amount of network resources are required.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] The present disclosure aims to provide a system that does not require a large amount of network resources even when collecting data from a large number of data sources.

Means for Solving the Problems

[0007] The system and method of the present disclosure are a system for collecting data from a plurality of data sources to a server and a method executed by the system, a network device for transferring data from the plurality of data sources to the server is defined for each data source, each of the plurality of data sources, when data to be transmitted to the server is generated, transmits a connection request for a communication path to the network device defined for its own device.

[0008] The data source device and method of the present disclosure are the data source device provided in a system for collecting data from a plurality of data source devices to a server and a method executed by the data source, wherein the data source device when data to be transmitted to the server is generated, transmits a connection request for a communication path to the network device defined for its own device.

[0009] Devices such as the controller and data source of the present disclosure can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided through a communication network. The program of the present disclosure is a program for realizing a computer as each functional part provided in the device according to the present disclosure, and is a program for causing a computer to execute each step provided in the method executed by the device according to the present disclosure.

Advantages of the Invention

[0010] According to the present disclosure, it is possible to provide a system that does not require a large amount of network resources even when collecting data from a large number of data sources.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These examples are merely illustrative, and the present disclosure can be implemented in various modified forms based on the knowledge of those skilled in the art. In the present specification and drawings, components with the same reference numerals indicate the same components as each other.

[0013] FIG. 1 shows an example of the system configuration of the present disclosure. The system of the present disclosure includes a plurality of data sources 10, a transmission control controller 20, and a server 40. The plurality of data sources 10, the transmission control controller 20, and the server 40 are connected by a communication network 80. The data source 10 is a terminal having user-side sensing data. The server 40 is a terminal that collects data from the data source.

[0014] The data source 10, the transmission control controller 20, and the server 40 of the present disclosure can also be realized by a computer and a program, and it is also possible to record the program on a recording medium or provide it through a communication network.

[0015] FIG. 2 shows an example of the configuration of the communication network 80. The communication network 80 is composed of a plurality of network devices 81. In the present embodiment, the network device 81 for transferring data from the data source 10 to the server 40 is determined for each data source 10. For example, the data source 10A and the server 40 are connected by network devices 81A and 81B, the data source 10B and the server 40 are connected by network devices 81A and 81C, and the data source 10C and the server 40 are connected by network devices 81D and 81E.

[0016] When each of the network devices 81A to 81E receives data from the data sources 10A to 10C destined for the server 40, it knows which network device to transfer the data to. Therefore, even if the data sources 10A to 10C do not know which network device 81 to use, if they specify the server 40 and send the data, the data will be sent to the server 40. From this, the present disclosure can transfer data losslessly and over a wide bandwidth when a communication path is generated.

[0017] In the present disclosure, in order to realize data transfer by RDMA, a method of allocating a lossless and wide-bandwidth communication path to the necessary data source 10 at the necessary timing is proposed. In the system of the present disclosure, when data to be transmitted to the server 40 is generated in each of the plurality of data sources 10, a communication path is generated at the transmission timing determined by the transmission policy of the own device using the network device 81 determined for the own device. Thereby, each data source 10 can autonomously transmit sensing data to the server 40 based on the transmission policy of the own device.

[0018] Examples of the transmission policy can be, for example, the following. First transmission policy: Generate a communication path immediately after data generation, perform data transfer, and release the communication path immediately after data transfer completion. Second transmission policy: Generate a communication path after a certain amount of data has accumulated, perform data transfer, and release the communication path immediately after data transfer completion. Third transmission policy: Generate a communication path immediately after data generation, perform data transfer, and continue to hold the communication path for a certain period of time even after data transfer completion. Fourth transmission policy: Generate a communication path after a certain amount of data has accumulated, perform data transfer, and continue to hold the communication path for a certain period of time even after data transfer completion.

[0019] In this embodiment, by providing the transmission control controller 20, a transmission policy that meets the data transfer requirements of the data source 10 is distributed. Then, according to the transmission policy considering the sensor characteristics and the data transfer requirements for each application, each data source 10 autonomously transmits sensing data. Thereby, this embodiment controls the trade-off between the immediacy of data transmission and the reduction of communication path utilization time.

[0020] Here, if the controller collectively controls the generation of paths as well, when a large number of path generation requests are made, the load on the controller increases, and it may become impossible to generate communication paths, or it may take longer than usual to generate them. Therefore, in the present disclosure, the data source 10 directly transmits a connection request for a communication path to the network devices 81A to 81E. By reflecting the settings received by the network devices 81A to 81E, the labor required to generate the communication path can be omitted. Thereby, the present disclosure can set the communication path in a shorter time. This will be described in detail below.

[0021] FIG. 3 shows a configuration example of the server 40. The server 40 includes a data reception function 41, a memory 42, and an application 43. The data reception function 41 receives sensing data from each data source 10. The memory 42 stores sensing data from the sensor 50. The application 43 collects sensing data from the sensor 50. The application 43 is an arbitrary application that collects arbitrary sensing data detected or generated by a user terminal, a sensor, an in-vehicle system, or the like.

[0022] FIG. 4 shows a configuration example of the data source 10. The data source 10 includes a requirement notification function 11, a transmission policy reception function 12, a data transmission timing control function 13, a communication path setting function 14, a data transmission function 15, a communication path release function 16, a generated data storage function 17, a requirement table 18, and a transmission policy table 19.

[0023] The generated data storage function 17 stores the sensing data from the sensor 50. The requirement notification function 11 reads the data transfer requirements from the requirement table 18 and notifies the transmission control controller 20 of the read data transfer requirements. The transmission policy receiving function 12 receives the transmission policy from the transmission control controller 20. The data transmission timing control function 13 controls the communication path setting function 14, the data transmission function 15, and the communication path release function 16 according to the transmission policy table 19. The communication path setting function 14 sends a connection request for generating a communication path with the server 40 to the network device 81. The data transmission function 15 transmits the sensing data stored in the generated data storage function 17 to the server 40. The communication path release function 16 sends a release request for releasing the communication path with the server 40 to the network device 81. The requirement table 18 stores the data transfer requirements for each data source 10. The transmission policy table 19 stores the policy for transmitting the sensing data from the data source 10 to the server 40.

[0024] FIG. 5 shows an example of the information stored in the requirement table 18 of each data source. In the requirement table, as data transfer requirements, the allowable delay time, the data generation frequency, and the data generation amount of each data source 10 are stored in association with the application. The allowable delay time is the allowable delay time from when the data is generated until it reaches the server. The data generation amount is the amount of data generated at once by the sensor 50.

[0025] FIG. 6 shows a configuration example of the transmission control controller 20. The transmission control controller 20 includes a requirement receiving function 21, a transmission policy determination function 22, a transmission policy distribution function 23, and a transmission policy determination rule 24.

[0026] FIG. 7 shows an example of the operation in the transmission control controller 20. Each data source 10 notifies the transmission control controller 20 of data transfer requirements based on the information in the requirement table 18 (S101). The data transfer requirements include the allowable delay time and data generation frequency of each data source 10 as shown in FIG. 5. The requirement receiver function 21 of the transmission control controller 20 receives the data transfer requirements from the data source 10. The transmission policy determination function 22 determines a transmission policy for each data source 10 according to the data transfer requirements in accordance with the transmission policy determination rule 24 (S102). The transmission policy distribution function 23 of the transmission control controller 20 distributes the determined transmission policy to each data source 10 (S103).

[0027] FIG. 8 shows an example of the operation in the transmission policy determination function 22. When the transmission control controller 20 acquires the data transfer requirements from the data source 10 (S11), it determines the requirement for the allowable delay time (S12). If the requirement for the allowable delay time is 100 ms or less (Yes in S12), it is determined to adopt a method of generating and transmitting a path immediately after the data is generated (S13). On the other hand, if the requirement for the allowable delay time exceeds 100 ms (No in S12), it is determined to adopt a method of accumulating a certain amount of sensing data and then transmitting it (S14). Next, the transmission control controller 20 determines the data generation frequency (S15). If the data generation frequency is 4 times / s or less (Yes in S15), it is determined to adopt a method of releasing the path immediately after the data is transmitted (S16). On the other hand, if the data generation frequency exceeds 4 times / s (No in S15), it is determined to adopt a method of waiting for a certain period of time after the data transmission is completed and then releasing the path (S17).

[0028] For example, in the case of the data transfer requirements shown in FIG. 5, the transmission policy determination function 22 determines as follows. · Data source A: Since the set time of the allowable delay time is 1000 ms and the set value of the data generation frequency is 4 times, the transmission control controller 20 determines the transmission policy as the second transmission policy. · Data source B: Since the set time of the allowable delay time is 200 ms and the set value of the data generation frequency is 10 times, the transmission control controller 20 determines the transmission policy to be the fourth transmission policy. · Data source C: Since the set time of the allowable delay time is 100 ms and the set value of the data generation frequency is 5 times, the transmission control controller 20 determines the transmission policy to be the third transmission policy. Thereby, a transmission policy as shown in FIG. 9 is given to data sources A to C.

[0029] Here, the data transfer requirement may include application information. In this case, the transmission control controller 20 sets a threshold value in steps S12 and S15 in consideration of the requirements included in the application information.

[0030] When the data source 10 receives a transmission policy from the transmission control controller 20, it stores it in the transmission policy table 19. Thereby, as shown in FIG. 9, a transmission policy suitable for each data source 10 is stored in the transmission policy table 19 of each data source 10.

[0031] When sensing data is generated, the data source 10 transmits a connection request for a communication path to the network device 81. The received device 81 reflects the setting. For example, when the data source 10A transmits sensing data, the data source 10A transmits a connection request for a communication path to the network device 81B. Then, the network device 81B generates a communication path with the data source 10A.

[0032] For each network device 81 set in the transmission route of the sensing data, the connection destination of the communication path of the sensing data is set in advance. Thus, in the present disclosure, predetermined network devices 81B and 81A that connect between the data source 10A and the server 40 generate a communication path between the data source 10A and the server 40. The communication path can use any means capable of transmitting the sensing data, and may be virtual such as a VLAN or physical such as an optical path.

[0033] For the connection request of the communication path, data in any format readable by the network device 81 can be used. For example, the Attribute ID field provided in the header of the RDMA (Remote Direct Memory Access) packet can be used.

[0034] FIG. 10 shows an example of the Attribute ID field. When using RoCE (RDMA over Converged Ethernet) v2 used in the UDP (User Datagram Protocol), the network device 81 can identify the negotiation based on the description in the Attribute ID field in the MAD Header. The MAD Header is stored in the DATH Header in the Base Transport Header.

[0035] The negotiation includes, for example, ConnectRequest for making an RDMA connection request, ConnectReply indicating an RDMA connection response, ReadyToUse indicating a usable state, DisconnectRequest for making an RDMA release request, and DisconnectReply indicating an RDMA release response. The value of the Attribute ID is determined for each negotiation. For example, if the Attribute ID is 0x0010, it can be used as a trigger for path generation in the network device 81B.

[0036] FIG. 11 shows an example of a sequence when generating and releasing a communication path. When using RDMA, data source 10A triggers a ConnectRequest to generate a communication path to server 40. At this time, network device 81B transfers the ConnectRequest to the next network device 81A. After the generation of the communication path is completed, network device 81A also transfers the ConnectRequest to server 40. Server 40 uses the generated communication path to send a ConnectReply to data source 10A. When data source 10A receives the ConnectReply from server 40, it sends a ReadyToUse to server 40. Thereby, RDMA communication from data source 10A to server 40 becomes possible.

[0037] When releasing the communication path, it is the same as when generating the communication path. Specifically, data source 10A triggers a Disconnect Request to release the communication path to server 40. At this time, when the release of the communication path is completed, network device 81B transfers the Disconnect Request to network device 81A. The same applies to network device 81A. When server 40 receives the Disconnect Request, it sends a DisconnectReply to data source 10A.

[0038] In network devices 81B and 81A, the timing of transferring the ConnectRequest is, for example, after the completion of the generation of the communication path. However, the present disclosure is not limited thereto. For example, network devices 81B and 81A may transfer the ConnectRequest without waiting for the completion of the generation of the communication path.

[0039] FIG. 12 shows an example of a sequence when generating and releasing a communication path. In this example, network devices 81B and 81A transfer a ConnectRequest without waiting for the completion of the generation of the communication path. In this case, network devices 81B and 81A are equipped with a function of sending a path setting completion notification to data source 10A which is the source of the ConnectRequest. In the present disclosure, since the transmission route of the sensing data is predetermined, data source 10A can confirm that the communication path has been set in all network devices 81B and 81A of the transmission route by counting the number of path setting completion notifications sent from network devices 81B and 81A. Once this confirmation is made, data source 10A sends ReadyToUse to server 40. Thereby, RDMA communication from data source 10A to server 40 becomes possible.

[0040] When releasing the communication path, data source 10A sends a Disconnect Request to server 40. Server 40 releases the communication path and sends a DisconnectReply to data source 10A. Since the communication path is released at server 40, the communication path is also released in network devices 81A and 81B.

[0041] FIG. 13 shows an example of transmission of sensing data from data source 10B to server 40. Data source 10B transmits sensing data according to the fourth transmission policy. Data source 10B accumulates the sensing data generated by sensor 50, and when the data reaches 20 MB, it generates a communication path with server 40 (S202). When the generation of the communication path is completed, data source 10B transmits sensing data to server 40 (S204-1, S204-2). After the transmission of the sensing data to server 40 is completed, data source 10B waits for 50 ms to send a request to release the communication path. If sensing data is generated during that time, the sensing data is transmitted each time (S204-3). When 50 ms has elapsed since the last data transmission (step S204-3), data source 10B releases the communication path (S205).

[0042] FIG. 14 shows an example of transmission of sensing data from data source 10A to server 40. Data source 10A transmits the sensing data according to the second transmission policy. Data source 10A accumulates the sensing data generated by sensor 50, and when the data reaches 15 MB, it generates a communication path with server 40 (S302). Then, when the generation of the communication path is completed, data source 10A transmits the accumulated sensing data to server 40 all at once (S304-1, S304-2, S304-3). When the transmission of the sensing data to server 40 is completed, data source 10A releases the communication path (S305).

[0043] FIG. 15 shows an example of transmission of sensing data from data source 10C to server 40. Data source 10C transmits the sensing data according to the third transmission policy. When sensing data is generated, data source 10C generates a communication path with server 40 (S402). When the generation of the communication path is completed, data source 10C transmits the sensing data to server 40 (S404-1), and then waits for 20 ms for the transmission of a communication path release request. If sensing data is generated during that time, the sensing data is transmitted each time (S404-2, S404-3). When 20 ms has elapsed since the last data transmission (step S404-3), data source 10C releases the communication path (S405).

[0044] As shown in FIGS. 13 to 15, in this embodiment, by allocating the communication path in a time-division manner at the necessary timing, depletion of network resources is prevented, and RDMA communication with a large number of data sources is realized. Since the communication path is allocated only for the period necessary for data source 10 that wants to transmit sensing data, the utilization rate of the path can be improved and network resources can be reduced.

[0045] (Effect of the present disclosure) As described above, since the present disclosure determines a transmission policy for each data source 10 and generates a communication path for each data source 10 according to the transmission policy, the communication path can be allocated in time division at the necessary timing. Therefore, the present disclosure can control, with respect to the trade-off between the immediacy of data transmission and the reduction of communication path utilization time, to reduce the communication path utilization time from each data source 10 to the server 40, facilitate the reuse of communication paths, and reduce the required number of communication paths for the entire communication network 80.

[0046] Therefore, the present disclosure can realize a lossless and broadband communication network 80 with communication paths, without requiring a large amount of network resources and enabling RDMA communication with a large number of data sources. Furthermore, since the present disclosure can prevent data loss in the communication network 80, reliable data transfer using RDMA can be realized.

[0047] In addition, in the present embodiment, an example where the data collected by the server 40 is sensing data is shown, but the present disclosure is applicable to any data for which collection is requested, such as user terminals, sensors, in-vehicle systems, etc.

[0048] Also, in the above-described embodiment, an example of setting a lossless and broadband communication path between the data source 10 and the server 40 is shown, but a data transfer path with unguaranteed quality may be set as the communication path.

Industrial applicability

[0049] The present disclosure can be applied to the information and communication industry.

Explanation of reference numerals

[0050] 10, 10A, 10B, 10C: Data source 20: Transmission control controller 40: Server 50: Sensor 11: Requirement Notification Function 12: Transmission Policy Receiving Function 13: Data Transmission Timing Control Function 14: Communication Path Setting Function 15: Data Transmission Function 16: Communication Path Release Function 17: Generated Data Storage Function 18: Requirement Table 19: Transmission Policy Table 21: Requirement Receiving Function 22: Transmission Policy Decision Function 23: Transmission Policy Distribution Function 24: Transmission Policy Decision Rule 41: Data Receiving Function 42: Memory 43: Application 80: Communication Network 81A, 81B, 81C, 81D, 81E: Network Devices

Claims

1. In a system for collecting data from a plurality of data sources to a server, a network device for transferring data from the plurality of data sources to the server is defined for each of the data sources, each of the plurality of data sources, when data to be transmitted to the server is generated, transmits a connection request for a communication path to the network device defined for its own device, the network device that has received the connection request for the communication path sequentially generates a predetermined communication path to the server according to the data source that is the source of the connection request for the communication path, a system.

2. The network device, when receiving a connection request for a communication path from a predetermined data source or network device, generates a communication path with the source of the connection request for the communication path, transmits a connection request for a communication path to a network device or server defined according to the source of the connection request for the communication path, and generates a communication path with the destination of the connection request for the communication path, The system according to claim 1.

3. The connection request for the communication path is transmitted using the header of an RDMA (Remote Direct Memory Access) packet, and a communication path enabling data transfer using RDMA is generated between the data source and the server, The system according to claim 1 or 2.

4. The connection request for the communication path is transmitted to the network device using the Attribute ID area provided in the header of the RDMA packet, The system according to claim 3.

5. A method executed by a system for collecting data from a plurality of data sources to a server, wherein the system has a network device for transferring data from the plurality of data sources to the server defined for each of the data sources, each of the plurality of data sources, when data to be transmitted to the server is generated, transmits a connection request for a communication path to the network device defined for its own device, the network device that has received the connection request for the communication path sequentially generates a predetermined communication path to the server according to the data source that is the source of the connection request for the communication path, a method.

6. The data source device provided in a system for collecting data from a plurality of data source devices to a server, A network device for transferring data to the server is predetermined, when data to be transmitted to the server is generated, a connection request for a communication path is transmitted to the predetermined network device determined for the own device, thereby generating a predetermined communication path from the own device to the server, when a connection response from the server is received, data is transmitted to the server using the communication path generated by the network device, data source device.

7. A method executed by the data source device provided in a system that collects data from a plurality of data source devices to a server, the data source device, when data to be transmitted to the server is generated, a connection request for a communication path is transmitted to the network device determined for the own device, thereby generating a predetermined communication path from the own device to the server, when a connection response from the server is received, data is transmitted to the server using the communication path generated by the network device, method.

8. A program for realizing a computer as the data source provided in the system according to any one of Claims 1 to 4.

Images