Communication system and communication method

Abstract

The purpose of the present invention is to provide an appropriate communication system for transmitting high-quality video data in real time when transmitting data from a client to a server using a plurality of streams in parallel.　One of typical communication systems according to the present invention is a communication system that transmits and receives data using a plurality of streams in parallel between a server and a client, wherein, if the transmission time per stream is on an increasing trend, the client sets a new standby time by performing a process to increase the standby time, and if the transmission time per stream is not on an increasing trend, the client newly calculates and sets a fixed value by performing a process of fixation of the standby time on the basis of data transmission statistical information.

Description

Communication system and communication method

[0001] The present invention relates to a communication system and a communication method.

[0002] In recent years, the use of drones has rapidly expanded, and the use of video data captured by drone-mounted cameras has also progressed. Video information captured by drone-mounted cameras is generally transmitted wirelessly in real time and used by pilots to remotely control the drone. It may also be subjected to various analyses and processing as needed. For this reason, video data transmitted from drones must be high-quality data that is free of image distortion or jitter, in addition to being transmitted in real time.

[0003] Typically, video data captured by cameras mounted on drones is often transmitted using TCP communication. However, transmitting data via TCP communication poses the issue of potential Head of Line (HoL) blocking. HoL blocking is a phenomenon in which the arrival of subsequent data is delayed due to a delay in the first data. HoL blocking occurs when the retransmission control required due to packet loss delays the arrival of subsequent packets. HoL blocking is particularly susceptible to affecting video quality, causing a time lag between the video on-site and the video on the data receiving side, and can lead to a decrease in quality due to video jerking.

[0004] Non-Patent Document 1 discloses a method for transmitting video data in a multi-stream communication system using QUIC / WebTransport. In the above-mentioned QUIC / WebTransport, if packet loss occurs in one stream, retransmission control is performed for each stream. This is believed to be able to suppress HoL blocking to some extent. Furthermore, it is believed that communication bandwidth can be used efficiently by lowering the transmission priority of a stream for which retransmission control has occurred, or by canceling the transmission of a QUIC stream for unnecessary data. Patent Document 1 also proposes using multiple TCP sockets to transmit RTP (Real-time Transport Protocol) packets while rotating these TCP sockets. The technology discloses a technique for avoiding HoL blocking by managing the sequence numbers of RTP packets whose arrival has been confirmed while checking the confirmation responses of each TCP socket, and when a gap in the sequence numbers is detected, deleting the corresponding TCP socket and canceling the transmission of RTP packets that took a long time to resend.

[0005] Luke Curley and three others, "Live Media TranSport over QUIC," [Warp], March 13, 2023 <URL: https: / / www.ietf.org / archive / id / draft-lcurley-warp-04.html>

[0006] US Patent Application Publication No. 2020 / 259881

[0007] The above-mentioned prior art claims that it is possible to avoid HoL blocking by using multiple QUIC streams or TCP sessions, but does not disclose a method for appropriately setting the parallelism of QUIC streams or TCP sessions. Therefore, setting an excessive parallelism may unnecessarily strain the communication bandwidth, reducing the available communication bandwidth per stream or session and increasing latency. Conversely, setting too little parallelism may result in reduced bandwidth utilization efficiency. Similarly, the prior art also discloses a method for halting the transmission of QUIC streams or TCP sessions when data is determined to be old, but does not disclose a method for determining the threshold for determining old data. Therefore, if the threshold for determining old data is too small, transmission halts may occur frequently, reducing the FPS seen by the application and degrading video quality. Furthermore, if the threshold is too large, the transmission halt process does not occur, preventing the function from operating and resulting in no benefits.

[0008] Therefore, an object of the present invention is to provide a communication system suitable for transmitting high-quality video data in real time when transmitting data from a client to a server using multiple streams in parallel.

[0009] In order to solve the above problem, one representative communication system of the present invention is a communication system in which data is sent and received between a server and a client using multiple streams in parallel, and if the transmission time per stream is on an increasing trend, the client performs a process to increase the waiting time and set a new waiting time, and if the transmission time per stream is not on an increasing trend, the client performs a process to fix the waiting time based on data transmission statistical information and calculates and sets a new fixed value.

[0010] According to the present invention, it is possible to provide a communication system suitable for transmitting high-quality video information in real time when transmitting data from a client to a server using multiple streams in parallel. Problems, configurations, and effects other than those described above will become apparent from the description of the following embodiments of the invention.

[0011] FIG. 1 shows an example of a communication system according to a first embodiment. FIG. 2 is a diagram showing an example of the hardware configuration of a server. FIG. 3 is a sequence diagram illustrating communication processing in the communication system. FIG. 4 is a sequence diagram illustrating communication processing in the communication system when packet loss occurs. FIG. 5 is a flowchart illustrating processing for calculating a waiting time. FIG. 6 is a flowchart of processing for calculating a fixed value of waiting time. FIG. 7 is a diagram showing log information related to data communication between a client and a server. FIG. 8 shows an example of a communication system according to a second embodiment. FIG. 9 is a sequence diagram related to processing for changing settings of sensor data.

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to these embodiments. In the drawings, identical parts are denoted by the same reference numerals. When multiple components having the same or similar functions are present, the same reference numerals may be denoted by different subscripts. Furthermore, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted. Furthermore, terms such as "first," "second," and "third" may be used to describe various elements or components in this disclosure, but it should be understood that these elements or components should not be limited by these terms. These terms are used only to distinguish one element or component from another. Therefore, a first element or component discussed below may also be referred to as a second element or component without departing from the teachings of the inventive concept. The arrangement, number, range, etc. of each component shown in the drawings may not represent the actual arrangement, number, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the arrangement, number, range, etc. disclosed in the drawings.

[0013] In this disclosure, the term "stream" refers to a continuous flow of divided data in a communication method in which received data is divided and transmitted in sequence without waiting for the entire data to be received when transmitting data between a data source and a data destination. The term "standby time" refers to the wait time between transmitting data in one stream and transmitting data in the next stream. The term "data transmission statistical information" refers to information including the data transmission time required to transmit a given piece of data, calculated based on the number of streams used in previous data transmissions and the transmission time required for the data transmission. The term "network requirements" refers to the network processing capabilities required for data transmission, including, for example, acceptable latency, throughput, packet loss, etc. In this disclosure, acceptable latency is also referred to as the acceptable network delay time. The term "application requirements" refers to the tolerance of the response to processes such as conversion, sampling, and compression of sensor data performed by an application, and can be specified, for example, by frames per second (FPS). "Monitoring information" is information that quantitatively represents trends such as network bandwidth usage, number of connected clients, and number of violations of network requirements. "Planning information" is information about plans regarding client operations, including, for example, the planned operating area in which client drones will operate, the planned number of units to operate, and the planned operating hours.

[0014] First Embodiment First, a communication system 1 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 shows an example of the communication system 1. The communication system 1 mainly includes a client 100, a server 200, and a network 300, and transmits and receives data between the server 200 and the client 100 using multiple streams in parallel.

[0015] <Client> The client 100 is a source that transmits data in the communication system 1. The client 100 mainly includes a sensor unit 101, a transmission application unit 102, and a multi-stream transmission processing unit 103.

[0016] <<Sensor Unit>> The sensor unit 101 has a function of acquiring information about the surroundings of the client 100 and generating sensor data to be transmitted. In this embodiment, the sensor unit 101 is a camera that can capture video, and generates video data about the surroundings of the client 100 as sensor data. Note that the sensor unit 101 may be a device other than a camera. For example, it may be a weather sensor or a radio wave sensor. In this case, the sensor data is meteorological information such as temperature, rainfall, wind direction and speed, and wireless radio wave intensity.

[0017] <<Transmission Application Unit>> The transmission application unit 102 is a functional unit that acquires sensor data from the sensor unit 101 and transmits the data to the server 200 via the multi-stream transmission processing unit 103. The transmission application unit 102 has at least functions such as converting, sampling, and compressing the sensor data, and may also have other functions that can be implemented in any programming language.

[0018] <<Multi-Stream Transmission Processing Unit>> The multi-stream transmission processing unit 103 has a function of establishing multiple streams with the multi-stream reception processing unit 203 (described later) and transmitting sensor data received from the transmission application unit 102 over the streams. The multi-stream transmission processing unit 103 mainly includes an API 104, a protocol processing unit 105, a stream control unit 106, and a network requirement management unit 107.

[0019] <<API>> The API (Application Programming Interface) 104 is a program for linking programs together. The API 104, for example, links the transmission application unit 102 with the protocol processing unit 105 and the stream control unit 106, and can operate the stream control unit 106 based on the control of the transmission application unit 102, or conversely, can operate the transmission application unit 102 based on a return value sent from the stream control unit 106. In other words, the functions provided by the API 104 include session establishment (connect) and data transmission (send). In addition to the transmission application unit 102, the API 104 can also link the stream control unit 106 with a reception application unit 202 (described later).

[0020] <<Protocol Processing Unit>> The protocol processing unit 105 provides functions of a communication protocol required for communication between the multi-stream transmission processing unit 103 and a multi-stream reception processing unit 203 (described later). The communication protocols for which the protocol processing unit 105 provides functions include, for example, TCP, UDP, QUIC, HTTP, and Web Transport, but may also be other communication protocols.

[0021] <<Stream Control Unit>> The stream control unit 106 is a functional unit that manages and controls the stream of the communication protocol provided by the protocol processing unit 105, based on the network requirements managed by the network requirement management unit 107. The stream control unit 106 also manages data transmission statistical information accumulated in conjunction with the management and control of the stream of the communication protocol.

[0022] <<Network Requirement Management Unit>> The network requirement management unit 107 has a function of holding the network requirements and application requirements notified from the transmission application unit 102, and a function of converting the application requirements into network requirements. The network requirements managed by the network requirement management unit 107 may be predetermined requirements, or the network requirement management unit 107 may calculate and determine network requirements that satisfy the application requirements.

[0023] <Server> The server 200 is a destination that receives data in the communication system 1. The server 200 mainly includes a reception application unit 202, a multi-stream reception processing unit 203, a display unit 204, and a storage unit 205.

[0024] <<Multi-Stream Reception Processing Unit>> The multi-stream reception processing unit 203 establishes multiple streams with the multi-stream transmission processing unit 103 and receives sensor data. The multi-stream reception processing unit 203 mainly includes an API 104, a protocol processing unit 105, a stream control unit 106, and a network requirement management unit 107.

[0025] <<Reception Application Unit>> The reception application unit 202 receives sensor data from the client 100 via the multi-stream reception processing unit 203 and processes the sensor data. The sensor data processing performed by the reception application unit 202 includes visualizing information, analyzing the information, and storing the data. For example, when video data is acquired for real-time video distribution for remote control of a drone, the reception application unit 202 performs processes such as visualizing the video data and storing the sensor data in the storage unit 205. The reception application unit 202 may also have other functions that can be implemented in any programming language.

[0026] <<Display Unit>> The display unit 204 transmits the sensor data received by the receiving application unit 202 to the input / output interface 1005 (described later) and displays the data on a display or the like. At this time, the display unit 204 may also display other information held by the communication system 1, such as information about the client 100, and the status of data transmission and reception between the client 100 and the server 200.

[0027] <<Storage Unit>> The storage unit 205 transmits and stores the sensor data to the HDD 1003 (described later). At this time, the storage unit 205 may store other information besides the sensor data, such as bandwidth information of the network 300, in the HDD 1003.

[0028] <Network> The network 300 is a computer network that enables communication between the client 100 and the server 200. The network 300 is, for example, a 5G network, the Internet, or a LAN, but may also be any other type of computer network.

[0029] Next, the hardware configuration of the server 200 will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example of the hardware configuration of the server 200. The server 200 mainly includes a CPU 1000, a ROM 1001, a RAM 1002, a HDD 1003, a network interface 1004, and an input / output interface 1005. The components of the server 200 are communicably connected to each other via a bus or the like. The server 200 may also be configured as a cluster in which multiple server chassis are regarded as a single server, or as a virtual server. The hardware of the client 100 may also have a configuration equivalent to that of the server 200.

[0030] <CPU> The CPU 1000 is a device that controls other devices and circuits within the computer and performs data calculations, etc. In this embodiment, it functions as a calculation device that executes programs installed on the HDD 1003. Furthermore, the CPU 1000 may execute some of the processes performed by executing the programs using other calculation devices, such as ASICs and FPGAs.

[0031] <ROM> ROM 1001 stores firmware and the like for server 200. When server 200 is started up, it reads the firmware from ROM 1001 and directly controls the hardware. ROM 1001 is a non-volatile storage medium and is a storage device that can only read the stored firmware, but it may also be a storage device that can rewrite the firmware.

[0032] <RAM> The RAM 1002 is loaded with programs to be executed by the CPU 1000 and temporarily stores data used when the programs are executed.

[0033] <HDD> The HDD 1003 stores programs executed by the CPU 1000. The HDD 1003 is a non-volatile storage medium, and is a storage device in which the stored programs can be rewritten. The HDD 1003 may also be a device other than an HDD, such as an SSD.

[0034] <Network Interface> The network interface 1004 is an interface for connecting the server 200 to the network 300 .

[0035] <Input / Output Interface> The input / output interface 1005 is an interface for connecting to an operating device such as a keyboard and a mouse, and a display (display device).

[0036] <Communication Processing of Communication System> Next, the communication processing of the communication system 1 will be described with reference to Fig. 3. Fig. 3 is a sequence diagram illustrating the communication processing of the communication system 1. When the client 100 and the server 200 communicate with each other, a session is first established between the client 100 and the server 200, and a stream is established based on the session before data transmission. In the description of Fig. 3, the communication system 1 will be described on the assumption that sensor data is transmitted from the client 100 to the server 200, but the communication between the client 100 and the server 200 is not limited to the direction of transmission of the sensor data and the communication of the sensor data described here.

[0037] (Sequence S301) In sequence S301, the reception application unit 202 of the server 200 executes the API 104 to put the server 200 into a session establishment wait state. This puts the multi-stream reception processing unit 203 into a session establishment wait state. Then, when a session establishment request is received from the multi-stream transmission processing unit 103 of the client 100, the multi-stream reception processing unit 203 of the server 200 can establish a session, and a session is established between the client 100 and the server 200.

[0038] (Sequence S302) In sequence S302, the transmission application unit 102 of the client 100 executes the API 104 to establish a session with the server 200. As a result, a session establishment request is executed in the multi-stream reception processing unit 203, which is waiting for session establishment. At this time, the transmission application unit 102 specifies information such as the IP address of the server 200 and network requirements including an allowable network delay time as arguments of the API 104, and executes the session establishment request based on the information.

[0039] (Sequence S303) In sequence S303, the multi-stream reception processing unit 203 replies to the session establishment request from the multi-stream transmission processing unit 103, and a session is established.

[0040] (Sequence S304) In sequence S304, the reception application unit 202 enters a state of waiting to receive sensor data due to the establishment of a session.

[0041] (Sequence S305) In sequence S305, the transmission application unit 102 executes the API 104 and transmits sensor data to the multi-stream reception processing unit 203 with which a session has been established.

[0042] (Sequence S306) In sequence S306, the multi-stream transmission processing unit 103 of the client 100 calculates the waiting time and transmits the waiting time to the transmission application unit 102.

[0043] (Sequence S307) In sequence S307, the multi-stream reception processing unit 203 of the server 200 establishes the stream S and transmits the sensor data. At this time, the stream S ends when the transmission of the sensor data is completed.

[0044] (Sequence S308) In sequence S308, the multi-stream reception processing unit 203 of the server 200 receives the sensor data, and when the reception of the sensor data is completed, transmits a reception completion notification of the sensor data to the reception application unit 202.

[0045] In the subsequent sequences, after waiting for the waiting time specified in sequence S306, the processing of sequences S305 to S308 is repeated. At this time, for example, the response of the API 104 in the first sequence S305 is asynchronous with the response of the API 104 in the next sequence S305, and the transmission application unit 102 can transmit data in other streams S without waiting for the completion of data transmission of each stream S.

[0046] <Communication Processing When Packet Loss Occurs> Next, a case where packet loss occurs in the communication processing of the communication system 1 will be described with reference to Fig. 4. Fig. 4 is a sequence diagram illustrating the communication processing of the communication system 1 when packet loss occurs. Note that in this description, a case where three streams (stream S1, stream S2, and stream S3 in the order of establishment) are established and sensor data is transmitted will be described, and packet loss occurs in the second stream (stream S2). However, the same applies when packet loss occurs in the other streams.

[0047] (Steps S305 to S308) Steps S305 to S308 are similar to the communication processing of the communication system 1, and therefore description thereof will be omitted.

[0048] (Sequence S401: Determining Stream Stop Threshold) In sequence S305, the transmission application unit 102 executes the API 104 for transmitting sensor data and starts transmission on stream S2. If a packet loss occurs, in sequence S401, the multi-stream transmission processing unit 103 retransmits the data that experienced the packet loss on stream S2. However, if retransmissions are repeated for a period longer than a predetermined period due to circumstances such as repeated packet losses, this retransmission process may cause network congestion and network delays. Furthermore, if repeated retransmissions delay data transmission, even if the data was successfully transmitted, the data may be unnecessary. For this reason, it is desirable to determine a threshold for retransmitting data, i.e., a stream stop threshold, and stop streams whose survival time has been extended. For this reason, the stream control unit 106 determines the stream stop threshold based on, for example, the allowable network delay time of the network requirements input in sequence S302 of FIG. 3.

[0049] (Sequence S402) In sequence S402, the multi-stream transmission processing unit 103 stops transmitting stream S2. Specifically, the stream control unit 106 acquires the time since the establishment of stream S2 and determines whether the time has exceeded the stream stop threshold. If the time since the establishment of stream S2 has exceeded the stream stop threshold, the sensor data transmitted using stream S2 is discarded and the transmission of the sensor data is canceled. If the stream stop threshold has not been exceeded, the data in which packet loss occurred is retransmitted.

[0050] The processing of sequence S402 enables efficient use of the line bandwidth of network 300 by canceling the transmission of data that is no longer necessary even if it is sent late, such as old video data when transmitting video in real time.

[0051] <Waiting Time Calculation Process> Next, a process for calculating a waiting time specified in the communication process of the communication system 1 will be described with reference to Fig. 5. Fig. 5 is a flowchart illustrating the waiting time calculation process. In the waiting time calculation process of Fig. 5, a case will be described in which a waiting time is set when at least one stream has been established and transmission of sensor data has started, and another stream is subsequently established. However, regardless of whether or not there is a preceding stream, the waiting time can be calculated and set in the same way in other cases.

[0052] (Step S501) In step S501, the stream control unit 106 acquires data transmission statistical information such as the number of streams currently being transmitted, the time required for data transmission, etc. Also, it acquires information on the waiting time of the latest stream currently being transmitted.

[0053] (Step S502) In step S502, the stream control unit 106 determines whether the waiting time of the latest stream is a fixed, valid waiting time. Here, a valid fixed waiting time means that the time since the waiting time was fixed is within a predetermined time range. If the fixed waiting time is valid, the process proceeds to step S505; if it is not valid, the process proceeds to step S505.

[0054] When such a fixed waiting time is effective, using the fixed time again has the advantage of requiring a short calculation time since it operates using a relatively simple algorithm.

[0055] (Step S503) In step S503, the stream control unit 106 compares the data transmission time of the preceding stream with the allowable network delay time, and determines whether the data transmission time has exceeded the allowable network delay time. Here, the data transmission time is the time from when the transmission of sensor data using the stream began, that is, the lifetime of the stream. The allowable network delay time is a threshold time for determining a delay in the transmission of sensor data; if the transmission of sensor data exceeds this time, it is considered that a network delay has occurred in the transmission of the sensor data. If the lifetime of the stream is shorter than the allowable network delay time, the process proceeds to step S504; if it is invalid, the process proceeds to step S506.

[0056] (Step S504) In step S504, if the stream lifetime is shorter than the network delay tolerance, that is, it is determined that the network has sufficient capacity, the stream control unit 106 performs processing to reduce the standby time, generate more streams per unit time, and increase the amount of data transmitted. There are various possible methods for reducing the standby time, but the stream control unit 106 may reduce the standby time by a fixed amount. This method reduces the computational load and increases throughput in a short period of time. Alternatively, the standby time may be reduced by 1 / N (N is a positive integer greater than or equal to 1). In this case, exponentially reducing the standby time suppresses the increase in throughput at the beginning of communication and prevents frequent congestion. This allows the stream control unit 106 to transmit data using more streams.

[0057] (Step S505) In step S505, the stream control unit 106 notifies the transmission application unit 102 of the calculated waiting time, and the calculation process ends.

[0058] (Step S506) If the stream lifetime is equal to or greater than the network delay tolerance, the stream control unit 106 determines whether the waiting time can be fixed in step S506. Specifically, the stream control unit 106 analyzes the trend in the data transmission time per stream based on the number of streams used in data transmission and the lifetime of the completed streams, and determines whether the waiting time can be fixed. For example, if the number of streams used in data transmission is increasing but the data transmission time per stream is not decreasing, it is likely that an external factor other than the number of streams is causing a decrease in available network bandwidth or packet loss. In such a case, it can be determined that the network margin is narrowing, and it is determined that the waiting time cannot be fixed. Conversely, if the stream lifetime and the network delay tolerance are trending at approximately the same level, it is determined that the waiting time can be fixed, and the calculation of a fixed value for the waiting time proceeds. In other words, if the waiting time cannot be fixed, the process proceeds to step S507. If the waiting time can be fixed, the process proceeds to step S508.

[0059] (Step S507) In step S507, the stream control unit 106 increases the waiting time. At this time, the stream control unit 106 may increase the waiting time by a fixed amount, or may increase the waiting time by M times (M is a positive integer greater than or equal to 1).

[0060] (Step S508) In step S508, the stream control unit 106 calculates a fixed value for the waiting time. The stream control unit 106 can set the waiting time calculated immediately before the data transmission time exceeds the allowable network delay time as the fixed value, for example. Alternatively, the stream control unit 106 can calculate the fixed value using a method for calculating a fixed value as shown in FIG. 6, which will be described later.

[0061] <Calculation Process of Fixed Value of Waiting Time> Next, the calculation process of the fixed value of waiting time will be described with reference to Fig. 6. Fig. 6 is a flowchart of the process of calculating the fixed value of waiting time. The process in Fig. 6 is premised on the process of calculating the fixed value of waiting time calculated in the process of step S508 in Fig. 5.

[0062] (Step S601) In step S601, the stream control unit 106 extracts, from the acquired data transmission statistical information, information necessary for calculating an estimated value of the bandwidth of the network 300. The information necessary for calculating an estimated value of the bandwidth of the network 300 includes, for example, the amount of data transmitted in previously completed data transmissions, the average lifetime per stream during the data transmissions, and the average number of streams used in the data transmissions.

[0063] (Step S602) In step S602, the stream control unit 106 calculates an estimate of the bandwidth of the network 300 based on the information extracted from the data transmission statistical information. Specifically, the stream control unit 106 calculates the data transmission time taken to transmit the data based on the following formula:

[0064] Next, the stream control unit 106 calculates an estimated value of the bandwidth of the network 300 based on the following formula:

[0065] (Step S603) In step S603, the stream control unit 106 calculates the required bandwidth per stream to satisfy the network requirements based on the following formula.

[0066] (Step S604) In step S604, the stream control unit 106 calculates the number of streams that can be used simultaneously in data transmission. Specifically, the stream control unit 106 calculates the number of streams that can be used simultaneously based on the following formula:

[0067] (Step S605) In step S605, the stream control unit 106 calculates the waiting time. Specifically, the waiting time is calculated based on the following formula.

[0068] Thus, while the processing described in step S508 of FIG. 5 and in FIG. 6 involves a relatively high computational load, it has the advantage of being able to set an appropriate waiting time even if the data size changes by estimating the available network bandwidth.

[0069] <Log Information> Next, log information related to data communication will be described with reference to Fig. 7. Fig. 7 is a diagram showing log information 5000 related to data communication between the client 100 and the server 200. When transmitting data from the client 100 to the server 200, the communication system 1 outputs a log indicating the status of the data communication. The logs output by the communication system 1 include, for example, log information L5001 indicating the status when a session is established, log information L5002 when the waiting time is varied, or log information L5003 when data transmission is stopped. In this embodiment, the stream control unit 106 of the server 200 generates the log information 5000, but it may also be generated by another component, for example, the stream control unit 106 of the client 100.

[0070] <Operations and Effects> The communication system 1 according to the first embodiment has been described above. The communication system 1 according to the present disclosure includes a multi-stream transmission processor 103 and a multi-stream reception processor 203. When setting a waiting time for a stream to be transmitted to the server 200, the client 100 determines the trend in the transmission time per stream. If the transmission time per stream is on the rise, the client 100 increases the waiting time using a predetermined method to set a new waiting time. If the transmission time per stream is not on the rise, the client 100 fixes the waiting time based on data transmission statistical information to calculate and set a new fixed value. By calculating the waiting time and adjusting the stream multiplicity based on the calculated waiting time, the client 100 can improve the utilization efficiency of network bandwidth, prevent congestion, and avoid HoL blocking by transmitting data via multiple streams.

[0071] As explained above, in order to transmit high-quality data in real time from client 100 to server 200, it is necessary to adjust the stream multiplicity and the stream transmission cancellation threshold. However, manual adjustment requires tuning for each network state and device performance, which takes time to develop. Therefore, this invention provides a function that automatically adjusts the stream multiplicity and the data transmission cancellation threshold based on requirement information input by the application.

[0072] Second Embodiment Next, a communication system 2 according to a second embodiment will be described with reference to FIG. 8 . FIG. 8 shows an example of the communication system 2. The communication system 2 according to the second embodiment differs from the first embodiment in that the transmission application unit 102 includes a sensor data setting management unit 108. The communication system 2 according to the second embodiment also differs from the first embodiment in that the server 200 includes a monitoring function unit 206, a planning information management unit 207, and a sensor data setting notification unit 208, is connected to at least two or more clients 100, and has an M:1 connection relationship between the server 200 and the clients 100 (M is a positive integer equal to or greater than 2). In the following description, components that are the same as or equivalent to those in the first embodiment described above are denoted by the same reference numerals, and their description will be simplified or omitted.

[0073] The communication system 2 is a system that mainly includes at least two or more clients 100, a server 200, and a network 300, and in which the clients 100 transmit data to the server 200 connected via the network 300.

[0074] <Client> The client 100 is a data sender in the communication system 2 that transmits and receives data using streams. The client 100 mainly includes a sensor unit 101, a transmission application unit 102 including a sensor data setting management unit 108, and a multi-stream transmission processing unit 103.

[0075] <<Sensor Data Setting Management Unit>> The sensor data setting management unit 108 changes the settings of the sensor data generated by the sensor unit 101 based on the sensor data setting notification received from the server 200. The sensor data settings include, for example, the resolution of the camera image and the sampling rate at which sensor data is collected. By changing these sensor data settings, it is possible to change the amount of sensor data generated by the sensor unit 101.

[0076] <Server> The server 200 is a destination that receives data in the communication system 2 that transmits and receives data using streams. The server 200 mainly includes a reception application unit 202, a multi-stream reception processing unit 203, a display unit 204, a storage unit 205, a monitoring function unit 206, a planning information management unit 207, and a sensor data setting notification unit 208.

[0077] <<Monitoring Function Unit>> The monitoring function unit 206 collects requirement information such as the number of connected clients 100, bandwidth usage information for the network 300, and arrival delays of transmitted data as data transmission statistical information for each time series.

[0078] <<Planning Information Management Unit>> The planning information management unit 207 manages planning information of the clients 100 connected to the server 200. Information about the clients 100 managed by the planning information management unit 207 includes, for example, the number of clients 100 connected to the server 200, the area in which the clients operate, and the time period.

[0079] <<Sensor Data Setting Notification Unit>> The sensor data setting notification unit 208 determines whether or not congestion or a delay in data arrival is likely to occur in the network 300, based on the data transmission statistical information collected by the monitoring function unit 206 and the planning information managed by the planning information management unit 207. If congestion or a delay in data arrival is likely to occur in the network 300, the sensor data setting notification unit 208 transmits a sensor data setting change notification to the sensor data setting management unit 108. The sensor data setting management unit 108 changes the setting of the sensor data acquired by the sensor unit 101 based on the setting change notification.

[0080] <Sensor Data Change Notification Processing> Next, the sensor data change notification processing will be described with reference to FIG. 9 . FIG. 9 is a sequence diagram related to the sensor data setting change processing. There are two patterns for the sensor data change notification processing: processing during planning and processing during operation. For example, in the case of a drone, planning refers to the formulation of a flight plan, and operation refers to the time when the drone is flying. First, processing during planning refers to the sending of a setting change notification before data is transmitted from the client 100 to the server 200. Processing during operation refers to the sending of a setting change notification while data is being transmitted from the client 100 to the server 200. In FIG. 9 , processing during planning is illustrated in the upper part of FIG. 9 and is described in sequences S701 to S703. Processing during operation is illustrated in the lower part of FIG. 9 and is described in sequences S801 to S803.

[0081] <<Processing at Planned Time>> When processing at planned time, the communication system 2 assumes that no data has been transmitted from the client 100 to the server 200 .

[0082] (Sequence S701) In sequence S701, the sensor data setting notification unit 208 acquires statistical information on the number of clients 100 transmitting data to the server 200, the operating area, and the network bandwidth usage per client 100, based on the planning information and data transmission statistical information managed by the planning information management unit 207.

[0083] (Sequence S702) In sequence S702, the sensor data setting notification unit 208 determines whether congestion will occur in the network 300 when the client 100 transmits data, based on the number of clients 100 transmitting data to the server 200, the operating area, and statistical information on the network bandwidth used per client 100, which were acquired in sequence S701. If congestion in the network 300 is predicted, setting values ​​for the clients 100 and the sensor unit 101 are calculated to suppress the amount of sensor data so that the overall amount of data transmitted over the network is at a level that does not cause congestion in the network 300.

[0084] (Sequence S703) In sequence S703, the sensor data setting notification unit 208 transmits the calculated setting value to the transmission application unit 102 and notifies the sensor data setting management unit 108 of a setting change notification. At this time, the client 100 changes the setting of the sensor unit 101 based on the setting change notification received by the sensor data setting management unit 108. In this case, if necessary to change the setting of the sensor unit 101, the sensor unit 101 may be restarted.

[0085] <<When processing during operation>> When processing during operation, the communication system 2 assumes that data is being transmitted from the client 100 to the server 200 and that the processing of sequences S301 to S308 in Figure 3 is being repeatedly executed.

[0086] (Sequence S801) In sequence S801, the sensor data setting notification unit 208 acquires data transmission statistical information in real time from the stream control unit 106. The data transmission statistical information referenced by the sensor data setting notification unit 208 may be acquired by extracting information based on specific requirements. For example, data transmission statistical information for each operating area may be acquired. Furthermore, plan information may be reflected in the data transmission statistical information.

[0087] (Sequence S802) In sequence S802, the sensor data setting notification unit 208 determines, based on the data transmission statistical information acquired in sequence S701, whether congestion will occur in the network 300 when the client 100 transmits data. If congestion is predicted to occur in the network 300, as in sequence S702, the setting change notification is calculated to suppress the amount of sensor data to be transmitted over the network so that the amount of data to be transmitted over the network is at a level that will not cause congestion in the network 300. If congestion is not predicted, the processing ends.

[0088] (Sequence S803) In sequence S803, the sensor data setting notification unit 208 transmits the calculated setting change notification to the transmission application unit 102, and notifies the sensor data setting management unit 108 of the setting change notification. At this time, the client 100 changes the setting of the sensor unit 101 based on the setting change notification received by the sensor data setting management unit 108. In this case, if necessary to change the setting of the sensor unit 101, the sensor unit 101 may be restarted.

[0089] <Actions and Effects> The communication system 2 according to the second embodiment has been described above. The communication system 2 according to the present disclosure includes a sensor data setting management unit 108 and a sensor data setting notification unit 208. Based on a setting change notification from the sensor data setting notification unit 208, the sensor data setting management unit 108 can change the settings of the sensor data generated by the sensor unit 101. This achieves the effects of the first embodiment, and in addition to the effects of the first embodiment, when there are multiple clients, it is possible to prevent congestion caused by a lack of bandwidth on the network 300 due to the data volume of the sensor data by controlling the amount of data transmitted to the network for each client. Furthermore, by controlling the amount of data transmitted from each client for each operating time period based on the client's operating area or planned information, rather than uniformly controlling multiple clients, it is possible to achieve accurate network operation according to more detailed conditions and transmission and reception of high-quality data.

[0090] As described above, the communication system 2 according to an embodiment of the present disclosure can automatically set the appropriate timing for issuing a transmission request and the transmission stop threshold according to application requirements in communication that avoids HoL blocking using multiple streams. This makes it possible to improve communication quality while reducing the amount of work required for tuning according to the application, network characteristics, and device performance. Furthermore, even when it is difficult to guarantee communication quality due to network congestion, the amount of data can be reduced according to the situation, and communication quality can be guaranteed by avoiding congestion.

[0091] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present invention.

[0092] The present invention can also take the following aspects. (Aspect 1) A communications system in which data is transmitted and received between a server and a client using multiple streams in parallel, wherein the client: if the transmission time per stream is on an increasing trend, performs processing to increase the wait time and set a new wait time; if the transmission time per stream is not on an increasing trend, performs processing to fix the wait time based on data transmission statistical information to newly calculate and set a fixed value. (Aspect 2) The communications system according to Aspect 1: if the wait time is fixed and a certain time has passed since it was fixed, resets the fixed wait time; if the wait time is not fixed or a certain time has passed since it was fixed, performs processing to increase and fix the wait time. (Aspect 3) The communications system according to Aspect 1 or 2: wherein the client: if the data transmission time of the preceding stream is shorter than the allowable network delay time, decreases the wait time. (Aspect 4) A communications system according to any one of Aspects 1 to 3, wherein the client includes a stream control unit that controls stopping of data transmission of the stream, wherein the stream control unit determines a stream stop threshold for stopping data transmission of the stream that satisfies network requirements based on the data transmission statistical information, and stops the data transmission of the stream when the transmission time of the data exceeds the stream stop threshold. (Aspect 5) A communications system according to Aspect 4, wherein the network requirements are specified by converting application requirements including FPS (Frames Per Second). (Aspect 6) A communications system according to any one of Aspects 1 to 5, wherein the data transmission statistical information is information including a data transmission time required to transmit predetermined data, calculated based on the number of streams used in data transmissions executed in the past and the transmission time required for the data transmission.(Aspect 7) A communication system according to any one of Aspects 1 to 6, wherein the client includes: a sensor unit that generates data to be transmitted; and a sensor data setting manager that controls a data volume of the data, wherein the sensor data setting manager changes the data volume based on monitoring information. (Aspect 8) A communication system according to Aspect 7, wherein the server receives the data from at least two or more of the clients, and the sensor data setting manager further changes the data volume based on planning information. (Aspect 9) A communication method according to any one of Aspects 1 to 8, comprising: a first step of increasing a wait time and setting a new wait time when the transmission time per stream is on an increasing trend; and a second step of fixing the wait time based on data transmission statistical information and calculating and setting a new fixed value when the transmission time per stream is not on an increasing trend.

[0093] 100: Client, 200: Server, 300: Network, 101: Sensor unit, 102: Transmission application unit, 103: Multi-stream transmission processing unit, 203: Multi-stream reception processing unit, 104: API, 105: Protocol processing unit, 106: Stream control unit, 107: Network requirement management unit, 108: Sensor data setting management unit, 202: Reception application unit, 204: Display unit, 205: Storage unit, 206: Monitoring function unit, 207: Planning information management unit, 208: Sensor data setting notification unit, 5000: Log information

Claims

1. A communications system in which data is transmitted and received between a server and a client using multiple streams in parallel, wherein the client, if the transmission time per stream is showing an increasing trend, performs a process of increasing the wait time and sets a new wait time, and if the transmission time per stream is not showing an increasing trend, performs a process of fixing the wait time based on data transmission statistical information and newly calculates and sets a fixed value.

2. A communication system as claimed in claim 1, wherein, if the waiting time is fixed and if a certain period of time has elapsed since the waiting time was fixed, the fixed waiting time is reset, and, if the waiting time is not fixed or if a certain period of time has elapsed since the waiting time was fixed, the communication system increases the waiting time and fixes it.

3. A communication system according to claim 1, wherein said client reduces said waiting time when the data transmission time of the preceding stream is smaller than the network delay tolerance time.

4. A communications system according to claim 1, wherein the client includes a stream control unit that controls the stopping of data transmission of the stream, and the stream control unit determines a stream stop threshold for stopping data transmission of the stream that satisfies network requirements based on the data transmission statistical information, and stops data transmission of the stream when the transmission time of the data exceeds the stream stop threshold.

5. A communication system according to claim 4, wherein the network requirements are defined by converting application requirements including FPS (Frame Per Second) into the network requirements.

6. A communication system according to claim 1, wherein the data transmission statistical information includes a data transmission time required to transmit a specified piece of data, the data transmission time being calculated based on the number of streams used in data transmissions performed in the past and the transmission time required for the data transmissions.

7. A communication system according to claim 1, wherein the client includes a sensor unit that generates data to be transmitted, and a sensor data setting management unit that controls a data volume of the data, and the sensor data setting management unit changes the data volume based on monitoring information.

8. A communication system according to claim 7, wherein the server receives the data from at least two or more of the clients, and the sensor data setting management unit further changes the data capacity based on planning information.

9. A communications method comprising: a first step of increasing a waiting time and setting a new waiting time when the transmission time per stream is showing an increasing trend; and a second step of fixing the waiting time based on data transmission statistical information and calculating and setting a new fixed value when the transmission time per stream is not showing an increasing trend.

Images