System, method and program for collecting data

The system optimizes data collection from multiple sources by dynamically allocating communication paths based on source-specific policies, ensuring efficient and reliable data transfer without excessive network resource use.

JP7582486B2Active Publication Date: 2024-11-13NIPPON TELEGRAPH & TELEPHONE CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023539402
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-08-02
Publication Date
2024-11-13
Estimated Expiration
2041-08-02

AI Technical Summary

Technical Problem

Conventional communication methods for data collection from multiple sources require significant network resources, leading to bottlenecks and inefficiencies.

Method used

A system that includes a controller to determine transmission policies for data sources, allowing data sources to autonomously request communication paths based on their requirements, and a path management controller to allocate paths efficiently, thereby optimizing network resource usage.

Benefits of technology

The system enables efficient, lossless, and wideband data transfer from multiple sources without excessive network resource consumption, improving path utilization and reducing the need for redundant paths.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007582486000001
    Figure 0007582486000001
  • Figure 0007582486000002
    Figure 0007582486000002
  • Figure 0007582486000003
    Figure 0007582486000003
Patent Text Reader

Abstract

The purpose of this disclosure is to provide a system in which large amounts of network resources are not required even when data is collected from a large number of data sources. This disclosure provides a system for collecting data from multiple data sources in a server. In the system, each data source transmits a data transfer requirement to a controller, the controller determines a transmission policy for the data source on the basis of the data transfer requirement, the data source autonomously transmits a communication path generation request on the basis of the transmission policy, the controller generates a communication path on the basis of the communication path generation request from the data source, and the data source transmits data using the generated communication path.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to communication control for data collection. [Background technology]

[0002] With the development of user terminals, sensors, and in-vehicle systems, the number of sensors and the amount of data generated by each sensor are increasing dramatically. When collecting such data, the communication processing on the receiving side can become a bottleneck in conventional communication methods.

[0003] RDMA (Remote Direct Memory Access) is being considered as a high-speed data transfer method. With RDMA, data is transferred from the memory of a local computer to the memory of a different remote computer using DMA (direct data transfer between peripheral devices and main memory (RAM) without going through the CPU). Because of this, RDMA does not require CPU processing for data transfer, and can avoid the bottleneck on the receiving side.

[0004] A remote transfer technology using this RDMA has been proposed (for example, see Non-Patent Document 1). However, Non-Patent Document 1 states that in order to transfer data using RDMA in a wide-area communication network, it is necessary to set a communication path for each flow in advance. Therefore, when collecting data from multiple data sources, a large amount of network resources is required. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] "One person, one wavelength" for ultra-low latency at 100Gbps, moving away from TCP / IP, Nikkei Electronics, 2021-03, P40-47 Summary of the Invention [Problem to be solved by the invention]

[0006] The present disclosure aims to provide a system that does not require a large amount of network resources even when collecting data from multiple data sources. [Means for solving the problem]

[0007] The disclosed system and method include: In a system that collects data from multiple data sources into a server, The system includes a controller that determines a transmission policy for disseminating data to the plurality of data sources; the data source sends to the controller data transfer requirements for transmitting data to the server; The controller determines a transmission policy for each data source based on the data transfer requirements; The data source autonomously transmits a communication path creation request based on the transmission policy; the controller generates a communication path from the data source to the server based on a communication path generation request from the data source; The data source transmits data using the created communication path.

[0008] The controller, data source, and other devices of the present disclosure may be realized by a computer and a program, and the program may be recorded on a recording medium or provided through a communication network. The program of the present disclosure is a program for causing a computer to realize each functional unit of the device according to the present disclosure, and is a program for causing a computer to execute each step of a method executed by the device according to the present disclosure. Effect of the Invention

[0009] According to the present disclosure, it is possible to provide a system that does not require a large amount of network resources even when collecting data from a large number of data sources. [Brief description of the drawings]

[0010] [Figure 1] 1 illustrates an example system configuration of the present disclosure. [Diagram 2] 1 shows an example of the configuration of a communication network. [Diagram 3] An example of a server configuration is shown. [Figure 4] An example of a data source configuration is shown below. [Diagram 5] An example of information stored in a requirements table of each data source is shown below. [Figure 6] 2 shows an example of the configuration of a transmission controller. [Figure 7] 13 shows an example of the operation of the transmission controller. [Figure 8] 13 shows an example of an operation of a transmission policy decision function. [Figure 9] 13 illustrates an example of information stored in a transmission policy table. [Figure 10] 1 shows an example of the configuration of a path management controller. [Figure 11] 13 shows an example of a path setting location table. [Figure 12] 13 shows an example of the operation of a path management controller. [Figure 13] 13 shows an example of the operation of a path management controller. [Figure 14] 13 shows an example of the operation of a path management controller. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings are assumed to indicate the same components.

[0012] (Summary of the Disclosure) 1 shows an example of a system configuration of the present disclosure. The system of the present disclosure includes a plurality of data sources 10, a transmission control controller 20, a path management controller 30, and a server 40. The plurality of data sources 10, the transmission control controller 20, the path management controller 30, and the server 40 are connected via a communication network 80. The data sources 10 are terminals having user-side sensing data. The server 40 is a terminal that collects data from the data sources.

[0013] In this embodiment, an example is shown in which the transmission control controller 20 and the path management controller 30 are separately arranged, but these controllers may be provided in a common device, or may be composed of multiple devices arranged in a distributed manner. The device of the present invention can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided through a communication network.

[0014] 2 shows an example of the configuration of a communication network 80. The communication network 80 is composed of a plurality of network devices 81. In this embodiment, the network devices 81 used between the data source 10 and the server 40 are determined in advance. For example, the data source 10A and the server 40 are connected by network devices 81A and 81B, the data source 10B and the server 40 are connected by network devices 81A and 81C, and the data source 10C and the server 40 are connected by network devices 81D and 81E.

[0015] When each of network devices 81A-81E receives data addressed to server 40 from data sources 10A-10C, it knows which network device to transfer the data to. Therefore, even if data sources 10A-10C do not know which network device 81 to use, if they specify server 40 and transmit data, the data is sent to server 40. Thus, the present disclosure makes it possible to transfer data losslessly and at a wide bandwidth when a communication path is generated.

[0016] In this disclosure, in order to realize data transfer by RDMA, a method is proposed in which a lossless and wideband communication path is assigned to a necessary data source 10 at a necessary timing. The system of the present disclosure comprises: A data source 10 sends a data transfer request to a transmission controller 20; The transmission controller 20 receives the data and determines a transmission policy for each data source 10 and distributes it to the data source 10. The data source 10 autonomously requests the path management controller 30 to generate a communication path based on a transmission policy; After establishing the communication path, the data source 10 transmits the sensing data to the server 40 .

[0017] The transmission policy can be exemplified as follows. First transmission policy: A communication path is generated immediately after data is generated, data is transferred, and the communication path is released immediately after the data transfer is completed. Second transmission policy: A communication path is generated and data is transferred after a certain amount of data has accumulated, and the communication path is immediately released after the data transfer is completed. Third transmission policy: A communication path is created immediately after data is generated, data is transferred, and the communication path is maintained for a certain period of time even after the data transfer is completed.

[0018] In this embodiment, by providing a transmission control controller 20, a transmission policy that matches the data transfer requirements of the data source 10 is distributed. Then, according to a transmission policy that takes into account the sensor characteristics and the data transfer requirements for each application, each data source 10 notifies a path creation request or release request to the path management controller 30, and controls the timing of path creation or release. In this way, this embodiment controls the trade-off between the immediacy of data transmission and the reduction of communication path usage time. This will be described in detail below.

[0019] 3 shows an example of the configuration of the server 40. The server 40 includes a data receiving function 41, a memory 42, and an application 43. The data receiving function 41 receives the sensing data from each data source 10 . The memory 42 stores the sensing data from the sensor 50 . The application 43 collects sensing data from the sensor 50. The application 43 is any application that collects any sensing data detected or generated by a user terminal, a sensor, an in-vehicle system, or the like.

[0020] 4 shows an example of the configuration of the data source 10. The data source 10 includes a requirement notification function 11, a transmission policy receiving function 12, a data transmission timing control function 13, a communication path setting function 14, a data transmission function 15, a communication path release function 16, a generated data storage function 17, a requirement table 18, and a transmission policy table 19.

[0021] The generated data storage function 17 stores the sensing data from the sensor 50 . The requirement notification function 11 reads out the data transfer requirements from the requirement table 18, and notifies the transmission controller 20 of the read out data transfer requirements. The transmission policy receiving function 12 receives a transmission policy from the transmission controller 20 . The data transmission timing control function 13 controls the communication path setting function 14 , the data transmission function 15 and the communication path release function 16 in accordance with the transmission policy table 19 . The communication path setting function 14 transmits a path creation request to the path management controller 30 . The data transmission function 15 transmits the sensing data stored in the generated data storage function 17 to the server 40 . The communication path release function 16 sends a path release request to the path management controller 30 . The requirements table 18 stores the data transfer requirements for each data source 10 . The transmission policy table 19 stores a policy for transmitting the sensing data from the data source 10 to the server 40 .

[0022] 5 shows an example of information stored in the requirement table 18 for each data source. In the requirement table, the allowable delay time, data generation frequency, and data generation amount of each data source 10 are stored as data transfer requirements, linked to the application. The allowable delay time is the allowable delay time from when data is generated until it reaches the server. The data generation amount is the amount of data generated at one time in the sensor 50.

[0023] 6 shows an example of the configuration of the transmission controller 20. The transmission controller 20 includes a requirement receiving function 21, a transmission policy determining function 22, a transmission policy distribution function 23, and a transmission policy determining rule 24.

[0024] FIG. 7 shows an example of the operation of the transmission controller 20. As shown in FIG. Each data source 10 notifies the transmission controller 20 of data transfer requirements based on the information in the requirements table 18. The data transfer requirements include the allowable delay time and data generation frequency of each data source 10, as shown in FIG. A requirement receiving function 21 of the transmission controller 20 receives data transfer requirements from the data source 10. A transmission policy determining function 22 determines a transmission policy for each data source 10 in accordance with the data transfer requirements, in accordance with transmission policy determining rules 24. The transmission policy distribution function 23 of the transmission controller 20 distributes the determined transmission policy to each data source 10 .

[0025] The transmission policy decision rule 24 determines a transmission policy according to the data transfer requirements for each data source 10 and each application. For example, the following can be given. (i) If the tolerable delay time is equal to or less than the set time and the data generation frequency is equal to or less than the set value, the above-mentioned first transmission policy is determined. (ii) If the tolerable delay time exceeds the set time and the data generation frequency is equal to or lower than the set value, the second transmission policy is determined. (iii) If the tolerable delay time is equal to or less than the set time and the data generation frequency exceeds the set value, the third transmission policy described above is determined.

[0026] FIG. 8 shows an example of the operation of the transmission policy decision function 22. In FIG. When the transmission controller 20 acquires data transfer requirements from the data source 10 (S11), it determines the requirements for the allowable delay time (S12). If the allowable delay time requirement is 20 ms or less (Yes in S12), it decides on a method of generating a path immediately after data generation and transmitting the data (S13). On the other hand, if the allowable delay time requirement is more than 20 ms (No in S12), it decides on a method of accumulating a certain amount of sensing data and transmitting it (S14). Next, the transmission controller 20 judges the data generation frequency (S15). If the data generation frequency is 30 times / s or less (Yes in S15), a method is decided to release the path immediately after data transmission (S16). On the other hand, if the data generation frequency is more than 30 times / s (No in S15), a method is decided to wait a certain time after data transmission is completed before releasing the path (S17).

[0027] For example, if the set value of the allowable delay time for applications A, B, and C is 20 ms and the set value of the data generation frequency is 20 times, the transmission control controller 20 determines the transmission policy of data source 10A to be the second transmission policy, determines the transmission policy of data source 10B to be the first transmission policy, and determines the transmission policy of data source 10C to be the third transmission policy.

[0028] Here, the data transfer requirements may include application information. In this case, the transmission controller 20 sets the threshold value in steps S12 and S15 taking into account the requirements included in the application information.

[0029] When the data source 10 receives the transmission policy from the transmission controller 20, it stores it in the transmission policy table 19. As a result, a transmission policy suitable for each data source 10 is stored in the transmission policy table 19 of each data source 10, as shown in FIG.

[0030] 10 shows an example of the configuration of the path management controller 30. The path management controller 30 comprises a path setting request receiving function 31, a path setting function 32, and a path setting location table 33.

[0031] The path setting request receiving function 31 receives a communication path creation request or release request from each data source 10 . The path setting function 32 creates or releases a communication path in response to a request from each data source 10 to create or release a communication path. The path setting location table 33 manages setting information (creation / release) of the communication paths from each data source 10 to the server 40 .

[0032] 11 shows an example of the path setting location table 33. The path setting location table 33 stores information on the network devices 81 that transfer the sensing data from each data source 10. For example, in the case of the data source 10A, the information on the network device 81 includes identification information of the network devices 81A and 81B that connect the data source 10A and the server 40, and setting information (generation / release) of the communication paths in the network devices 81A and 81B.

[0033] Fig. 12 shows an example of the operation of the path management controller 30. Fig. 12 shows a method of immediately generating and transmitting a path after sensing data is generated. When sensing data is generated, the data source 10B transmits a communication path creation request to the path management controller 30 (S201). When the path management controller 30 receives the communication path creation request, it sets the communication path between the network devices 81A and 81C and creates the communication path (S202). When the communication path creation is completed, the path management controller 30 transmits a communication path creation completion notification to the data source 10B (S203). When the data source 10B receives the communication path creation completion notification, it transmits sensing data to the server 40 (S204). When the data source 10B completes transmitting the sensing data to the server 40, it transmits a communication path release request to the path management controller 30 (S205). When the path management controller 30 receives the communication path release request, it releases the communication paths of the network devices 81A and 81C, and transmits a communication path release completion notification to the data source 10B indicating that the communication path release has been completed (S206).

[0034] Fig. 13 shows an example of the operation of the path management controller 30. Fig. 13 shows a method of accumulating a certain amount of sensing data and transmitting it. In this method, the data source 10A accumulates sensing data generated by the sensor 50, and when a certain amount of data is accumulated, it transmits a communication path creation request to the path management controller 30 (S301). Then, when the data source 10A receives a communication path creation completion notification (S303), it transmits the accumulated sensing data collectively to the server 40 (S304-1, S304-2, S304-3).

[0035] Fig. 14 shows an example of the operation of the path management controller 30. Fig. 14 shows a method of holding a path for a certain period of time after transmission. In this method, when the data source 10C receives a communication path generation completion notification (S303), it transmits sensing data to the server 40 (S404-1, S404-2) and waits for a certain period of time thereafter to transmit a request to release the communication path. If sensing data is generated during that time, it transmits the sensing data each time (S404-3, S404-4). When a certain period of time has elapsed since the data source 10C received the communication path creation completion notification, the data source 10C transmits a communication path release request to the path management controller 30 (S405).

[0036] 12 to 14, in this embodiment, a communication path is allocated in a time-sharing manner at the required timing, thereby preventing the exhaustion of network resources and realizing RDMA communication with a large number of data sources. A communication path is allocated to the data source 10 that wishes to transmit sensing data only for the period required, which improves the path utilization rate and reduces network resources.

[0037] (Effects of the present disclosure) As described above, the present disclosure determines a transmission policy for each data source 10 and generates a communication path for each data source 10 according to the transmission policy, so that the communication paths can be allocated in a time-sharing manner at the required timing. Therefore, the present disclosure can reduce the communication path usage time from each data source 10 to the server 40, facilitate the reuse of communication paths, and reduce the number of communication paths required for the entire communication network 81, in response to the trade-off between the immediacy of data transmission and the reduction in the communication path usage time.

[0038] Therefore, the present disclosure can realize a lossless and wideband communication network 81 with a communication path, and can enable RDMA communication with a large number of data sources without requiring a large amount of network resources. Furthermore, the present disclosure can prevent data loss in the communication network 81, and therefore can realize reliable data transfer using RDMA.

[0039] In the present embodiment, an example has been shown in which the data collected by the server 40 is sensing data, but the present disclosure is applicable to any data that is required to be collected, such as data from user terminals, sensors, and in-vehicle systems.

[0040] In the above embodiment, an example of setting a lossless and wideband communication path between the data source 10 and the server 40 has been described, but the present disclosure is not limited to this. That is, the path management controller 30 may set a transfer path for data whose quality is not guaranteed as a communication path. [Industrial Applicability]

[0041] The present disclosure can be applied to the information and communications industry. [Explanation of symbols]

[0042] 10, 10A, 10B, 10C: Data Source 20: Transmission control controller 30: Path management controller 40: Server 50: Sensor 11: Requirement notification function 12: Sending policy receiving function 13: Data transmission timing control function 14: Communication path setting function 15: Data transmission function 16: Communication path release function 17: Generated data storage function 18: Requirements table 19: Sending policy table 21: Requirement reception function 22: Transmission policy decision function 23: Sending policy distribution function 24: Transmission policy decision rule 31: Path setting request receiving function 32: Path setting function 33: Path setting location table 41: Data reception function 42: Memory 43: Application 81A, 81B, 81C, 81D, 81E: Network equipment,

Claims

1. In a system that collects data from multiple data sources into a server, a controller for determining a transmission policy for distribution to the plurality of data sources; the data source sends to the controller data transfer requirements for transmitting data to the server; The controller determines a transmission policy for each data source based on the data transfer requirements; The data source autonomously transmits a communication path creation request based on the transmission policy; the controller generates a communication path from the data source to the server based on a communication path generation request from the data source; The data source transmits data using the created communication path. system.

2. The transmission policy is: A communication path is created immediately after data is generated, data is transferred, and the communication path is released immediately after the data transfer is completed. A system that generates a communication path and transfers data after a certain amount of data has been accumulated, and releases the communication path immediately after the data transfer is completed; and A system that creates a communication path immediately after data is generated, transfers the data, and continues to hold the communication path for a certain period of time after the data transfer is complete. Including, The system of claim 1 .

3. The data transfer requirements include an allowable delay time and a data generation frequency, The controller determines a transmission policy for each data source based on an allowable delay time and a data occurrence frequency in each data source.

3. A system according to claim 1 or 2.

4. The controller: If the allowable delay time is equal to or less than a set time and the data generation frequency is equal to or less than a set value, a transmission policy is determined in which a communication path is generated immediately after the data is generated, data is transferred, and the communication path is released immediately after the data transfer is completed; If the allowable delay time exceeds the set time and the data generation frequency is equal to or less than the set value, a transmission policy is decided in which a communication path is generated and data is transferred after a certain amount of data is accumulated, and the communication path is immediately released after the data transfer is completed; If the allowable delay time is equal to or less than the set time and the data generation frequency exceeds the set value, a transmission policy is determined in which a communication path is generated immediately after the data is generated, data is transferred, and the communication path is maintained for a certain period of time even after the data transfer is completed. The system of claim 3.

5. A method performed by a system for collecting data from multiple data sources into a server, comprising: The system includes a controller that determines a transmission policy for disseminating data to the plurality of data sources; the data source sends to the controller data transfer requirements for transmitting data to the server; The controller determines a transmission policy for each data source based on the data transfer requirements; The data source autonomously transmits a communication path creation request based on the transmission policy; the controller generates a communication path from the data source to the server based on a communication path generation request from the data source; The data source transmits data using the created communication path. method.

6. A program for causing a computer to realize each of the functional units of the controller according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Network router and method and unit for assigning resource in exchange

    JP2003060691A

  • Network control method, controller, and program

    JP2018152675A

  • Load distribution system, load distribution method, device and program constituting load distribution system

    WO2011074630A1