Communication system and communication method

Abstract

To provide a communication system capable of transmitting high-quality video data in real-time when transmitting data from a client to a server using multiple streams in parallel.SOLUTION: A typical disclosed communication system is a communication system that sends and receives data using multiple streams in parallel between the server and the client. The client is configured so as to, when the transmission time per stream is increasing, set a new waiting time by increasing the waiting time, and when the transmission time per stream is not increasing, execute to fix the waiting time based on data transmission statistics information by calculating a new fixed value and set the same.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a communication system and a communication method. [Background technology]

[0002] In recent years, the use of drones has expanded rapidly, and the use of video data captured by drone-mounted cameras has also progressed. Video information captured by drone-mounted cameras is generally wirelessly transmitted in real time and used by pilots to remotely control the drone. In addition, various analyses and processing may be performed as necessary. For this reason, video data transmitted from drones is required to be high-quality data that is free of image distortion and jitter, as well as real-time.

[0003] Generally, video data captured by cameras mounted on drones is often transmitted using TCP communication. However, when transmitting data using TCP communication, there is an issue that HoL blocking (Head Of Line Blocking) can occur. HoL blocking is a phenomenon in which the arrival of subsequent data is delayed due to a delay in the first data. If retransmission control due to packet loss takes time, the arrival of the subsequent packets is delayed, which occurs. HoL blocking is particularly likely to affect video quality, causing a time lag between the video on-site and the video on the data receiving side, and can easily lead to a decrease in quality due to video stuttering.

[0004] Non-Patent Document 1 discloses a method for transmitting video data in a multi-stream format using QUIC / WebTranSport in a communication system using multi-streams. In the above QUIC / WebTransport, if a packet loss occurs in one stream, retransmission control is performed for each stream, which is said to be able to suppress HoL blocking to some extent. Furthermore, it is possible to use communication bandwidth efficiently by lowering the transmission priority of a stream for which retransmission control has occurred, or by canceling the transmission of a QUIC stream for data that is no longer needed. Patent Document 1 also proposes using multiple TCP sockets to transmit RTP (Real-time Transport Protocol) packets while rotating these TCP sockets.Then, by monitoring the acknowledgement of each TCP socket and managing the sequence numbers of RTP packets whose arrival has been confirmed, when a gap is detected in the sequence numbers, the corresponding TCP socket is deleted, and the transmission of the RTP packets that took a long time to resend is canceled, thereby avoiding HoL blocking. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Luke Curley, 3 others “Live Media TranSport over QUIC”, [Warp], March 13, 2023<URL:https: / / www.ietf.org / archive / id / draft-lcurley-warp-04.html> [Patent documents]

[0006] [Patent Document 1] US Patent Application Publication No. 2020 / 259881 Summary of the Invention [Problem to be solved by the invention]

[0007] The above-mentioned prior art claims that it is possible to avoid HoL blocking by using multiple QUIC streams or TCP sessions, but does not disclose a method for appropriately setting the parallelism of the QUIC streams or TCP sessions. For this reason, if an excessive degree of parallelism is set, it may unnecessarily strain the communication bandwidth, reducing the available communication bandwidth per stream or session and increasing latency.On the other hand, if the degree of parallelism is too low, it may result in a decrease in the efficiency of bandwidth usage. Similarly, the prior art also discloses a method of stopping transmission of a QUIC stream or TCP session when data is determined to be old, but does not disclose a method for determining the threshold for determining old data. Therefore, if the threshold for determining old data is too small, transmission stops will occur frequently, and the FPS seen by the application will decrease, which may result in a deterioration in video quality. In addition, if the threshold is too large, the transmission stop process will not occur, so the function will not work and no effect will be obtained.

[0008] Therefore, an object of the present invention is to provide a communication system suitable for transmitting high-quality video data in real time when transmitting data from a client to a server using multiple streams in parallel. [Means for solving the problem]

[0009] In order to solve the above problems, one representative communication system of the present invention is a communication system in which data is transmitted and received between a server and a client using multiple streams in parallel, in which, if the transmission time per stream is showing an increasing trend, the client performs a process of increasing the waiting time and sets a new waiting time, and, if the transmission time per stream is not showing an increasing trend, performs a process of fixing the waiting time based on data transmission statistical information and newly calculates and sets a fixed value. Effect of the Invention

[0010] According to the present invention, it is possible to provide a communication system suitable for transmitting high quality video information in real time when transmitting data from a client to a server using multiple streams in parallel. Problems, configurations and effects other than those described above will become apparent from the following description of the preferred embodiment of the invention. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 shows an example of a communication system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server. [Diagram 3] FIG. 3 is a sequence diagram illustrating a communication process of the communication system. [Figure 4] FIG. 4 is a sequence diagram illustrating a communication process of the communication system when a packet loss occurs. [Diagram 5] FIG. 5 is a flowchart illustrating the waiting time calculation process. [Figure 6] FIG. 6 is a flowchart of a process for calculating a fixed value of the waiting time. [Figure 7] FIG. 7 is a diagram showing log information related to data communication between a client and a server. [Figure 8] FIG. 8 shows an example of a communication system according to the second embodiment. [Figure 9] FIG. 9 is a sequence diagram relating to the process of changing the settings of the sensor data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiment. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals. When there are multiple components having the same or similar functions, they may be described by using the same reference numerals with different subscripts, or when there is no need to distinguish between these multiple components, the subscripts may be omitted. In addition, although terms such as "first," "second," and "third" may be used in the present disclosure to describe various elements or components, it will be understood that these elements or components should not be limited by these terms. These terms are used only to distinguish one element or component from another element or component. Thus, a first element or component discussed below can also be referred to as a second element or component without departing from the teachings of the inventive concept. In order to facilitate understanding of the invention, the arrangement, number, range, etc. of each component shown in the drawings may not represent the actual arrangement, number, range, etc. Therefore, the present invention is not necessarily limited to the arrangement, number, range, etc. disclosed in the drawings.

[0013] In this disclosure, "stream" refers to a continuous flow of divided data in a communication method in which received data is divided in sequence and transmitted / received between a data sender and a data destination without waiting for the entire data to be received. The "waiting time" is the time to wait between sending data in one stream and sending data in the next stream. In addition, "data transmission statistical information" refers to information including the data transmission time required to transmit specified data, calculated based on the number of streams used in previously performed data transmissions and the transmission time required for the data transmissions. Moreover, the term "network requirements" refers to the network processing capabilities required for data transmission, and includes, for example, tolerable latency, throughput, packet loss, etc. In this disclosure, the tolerable latency is also referred to as the network delay tolerance time. Moreover, the "application requirements" refer to the tolerance of the response to the processing such as conversion, sampling, and compression of sensor data performed by the application, and can be defined, for example, by FPS (Frames Per Second). Moreover, "monitoring information" is information that quantitatively represents trends such as the network bandwidth usage, the number of connected clients, and the number of violations of network requirements. In addition, "planned information" refers to information about plans regarding the operation of the client, and includes, for example, information such as the planned operating area in which the client's drones will operate, the planned number of drones to operate, and the planned operating time periods.

[0014] [First embodiment] First, a communication system 1 according to a first embodiment will be described with reference to FIG. FIG. 1 shows an example of a communication system 1. The communication system 1 mainly includes a client 100, a server 200, and a network 300, and transmits and receives data between the server 200 and the client 100 using multiple streams in parallel.

[0015] <Client> The client 100 is a source that transmits data in the communication system 1. The client 100 mainly includes a sensor unit 101, a transmission application unit 102, and a multi-stream transmission processing unit 103.

[0016] <<Sensor section>> The sensor unit 101 has a function of acquiring information about the surroundings of the client 100 and generating sensor data to be transmitted. In this embodiment, the sensor unit 101 is a camera capable of capturing video, and generates video data of the surroundings of the client 100 as sensor data. The sensor unit 101 may be a device other than a camera. For example, it may be a weather sensor or a radio wave sensor. In this case, the sensor data is meteorological information such as temperature, rainfall, wind direction and speed, and radio wave intensity.

[0017] <<Transmission application part>> The transmission application unit 102 is a functional unit that acquires sensor data from the sensor unit 101 and transmits the sensor data to the server 200 via the multi-stream transmission processing unit 103 . The transmission application unit 102 has at least functions such as conversion, sampling, and compression of sensor data, and may also have other functions that can be implemented in any programming language.

[0018] <<Multi-stream transmission processing unit>> The multi-stream transmission processing unit 103 has a function of establishing a plurality of streams with a multi-stream reception processing unit 203 (to be described later) and transmitting sensor data received from the transmission application unit 102 onto the streams. The multi-stream transmission processing unit 103 mainly includes an API 104 , a protocol processing unit 105 , a stream control unit 106 , and a network requirement management unit 107 .

[0019] < <api>> An API (Application Programming Interface) 104 is a program for linking programs with each other. The API 104, for example, can link the transmission application unit 102 with the protocol processing unit 105 and the stream control unit 106, and can operate the stream control unit 106 based on the control of the transmission application unit 102, or conversely, can operate the transmission application unit 102 based on a return value transmitted from the stream control unit 106. In other words, the functions provided by the API 104 are functions such as session establishment (connect) and data transmission (send). Furthermore, the API 104 can also link a receiving application unit 202 (to be described later) and a stream control unit 106 in addition to the transmitting application unit 102 .

[0020] <<Protocol processing unit>> The protocol processing unit 105 provides a communication protocol function required for communication between the multi-stream transmission processing unit 103 and a multi-stream reception processing unit 203, which will be described later. The communication protocols for which the protocol processing unit 105 provides functions are, for example, TCP, UDP, QUIC, HTTP, WebTransport, etc., but may be other communication protocols.

[0021] <<Stream control section>> The stream control unit 106 is a functional unit that manages and controls the stream of the communication protocol provided by the protocol processing unit 105 based on the network requirements managed by the network requirement management unit 107 . The stream control unit 106 also manages data transmission statistical information accumulated in association with the management and control of the stream of the communication protocol.

[0022] <<Network Requirements Management Division>> The network requirement management unit 107 has a function of holding the network requirements and application requirements notified from the transmission application unit 102, and a function of converting the application requirements into network requirements. The network requirements managed by the network requirement management unit 107 may be requirements determined in advance, or the network requirement management unit 107 may calculate and determine network requirements that satisfy the application requirements.

[0023] <Server> The server 200 is a destination in the communication system 1 that receives data. The server 200 mainly includes a reception application unit 202 , a multi-stream reception processing unit 203 , a display unit 204 , and a storage unit 205 .

[0024] <<Multi-stream reception processing unit>> The multi-stream reception processing unit 203 establishes a plurality of streams with the multi-stream transmission processing unit 103 and receives sensor data. The multi-stream reception processing unit 203 mainly includes an API 104 , a protocol processing unit 105 , a stream control unit 106 , and a network requirement management unit 107 .

[0025] <<Receiving application part>> The reception application unit 202 receives sensor data from the client 100 via the multi-stream reception processing unit 203, and processes the sensor data. The processing of sensor data performed by the reception application unit 202 includes visualizing information, analyzing, storing data, etc. For example, when video data is acquired for real-time video distribution for remote control of a drone, the reception application unit 202 performs processing such as visualizing the video data and storing the sensor data in the storage unit 205. In addition, the reception application unit 202 may also have other functions that can be implemented in any programming language.

[0026] <<Display section>> The display unit 204 transmits the sensor data received by the reception application unit 202 to an input / output interface 1005 (described later) and displays the data on a display or the like. At this time, the display unit 204 may also display other information held by the communication system 1, such as information about the client 100, the status of data transmission and reception between the client 100 and the server 200, and the like.

[0027] <<Storage section>> The storage unit 205 transmits the sensor data to a HDD 1003 (described later) for storage. At this time, the storage unit 205 may store in the HDD 1003 other information than the sensor data, such as bandwidth information of the network 300.

[0028] <Network> The network 300 is a computer network that configures the client 100 and the server 200 so that they can communicate with each other. The network 300 may be, for example, a 5G network, the Internet, a LAN, or other types of computer networks.

[0029] Next, the hardware configuration of the server 200 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a hardware configuration of the server 200. The server 200 mainly includes a CPU 1000 , a ROM 1001 , a RAM 1002 , a HDD 1003 , a network interface 1004 , and an input / output interface 1005 . The components of the server 200 are communicatively connected to one another via a bus or the like. The server 200 may be configured as a cluster in which multiple server cases are regarded as one server, or as a virtual server. The hardware of the client 100 may also have the same configuration as the server 200.

[0030] <cpu> The CPU 1000 is a device that controls other devices and circuits within the computer and performs data calculations. In this embodiment, the HDD 1003 functions as a computing device that executes programs installed in the HDD 1003 . Furthermore, part of the processing performed by the CPU 1000 executing a program may be executed by another computing device, such as an ASIC or an FPGA.

[0031] <rom> The ROM 1001 stores firmware for the server 200 and the like. At startup, the server 200 reads the firmware from the ROM 1001 and directly controls the hardware. The ROM 1001 is a non-volatile storage medium and is a storage device that can only read out the firmware stored therein, but may be a storage device that can rewrite the firmware.

[0032] <ram> The programs executed by the CPU 1000 are loaded into the RAM 1002, and data used when the programs are executed is temporarily stored therein.

[0033] <hdd> The HDD 1003 stores the programs executed by the CPU 1000 . The HDD 1003 is a non-volatile storage medium, and is a storage device in which the programs stored therein are rewritable. The HDD 1003 may be a device other than a HDD, for example, an SSD.

[0034] <Network interface> The network interface 1004 is an interface for connecting the server 200 to the network 300 .

[0035] <Input / output interface> The input / output interface 1005 is an interface for connecting to an operating device such as a keyboard and a mouse, and a display (display device).

[0036] <Communication processing in communication systems> Next, the communication process of the communication system 1 will be described with reference to FIG. FIG. 3 is a sequence diagram illustrating the communication process of the communication system 1. As shown in FIG. When the client 100 and the server 200 communicate with each other, first, a session is established between the client 100 and the server 200, and a stream is established based on the session before data transmission. In addition, in the explanation of Figure 3, the communication system 1 is explained under the assumption that sensor data is transmitted from the client 100 to the server 200, but the communication between the client 100 and the server 200 is not limited to the direction of transmission of the sensor data and the communication of the sensor data described here.

[0037] (Sequence S301) In sequence S301, the receiving application unit 202 of the server 200 executes the API 104 so that the server 200 waits for session establishment. As a result, the multi-stream reception processor 203 waits for a session to be established. Then, when a session establishment request is made from the multi-stream transmission processing unit 103 of the client 100 , the multi-stream reception processing unit 203 of the server 200 can establish a session, and a session is established between the client 100 and the server 200 .

[0038] (Sequence S302) In sequence S302, the transmission application unit 102 of the client 100 executes the API 104 to establish a session with the server 200. As a result, the session establishment request is executed in the multi-stream reception processing unit 203, which is waiting for the session establishment. At this time, the transmission application unit 102 specifies information such as the IP address of the server 200 and network requirements including the allowable network delay time as arguments to the API 104, and executes a session establishment request based on the information.

[0039] (Sequence S303) In sequence S303, the multi-stream reception processing unit 203 replies to the session establishment request from the multi-stream transmission processing unit 103, and the session is established.

[0040] (Sequence S304) In sequence S304, the receiving application unit 202 establishes a session. The device enters a state of waiting to receive sensor data.

[0041] (Sequence S305) In sequence S305, the transmission application unit 102 executes the API 104 and transmits sensor data to the multi-stream reception processing unit 203 with which a session has been established.

[0042] (Sequence S306) In sequence S306, the multi-stream transmission processing unit 103 of the client 100 calculates the waiting time, and transmits the waiting time to the transmission application unit .

[0043] (Sequence S307) In sequence S307, the multi-stream reception processing unit 203 of the server 200 establishes a stream S and transmits sensor data. At this time, the stream S ends when the transmission of the sensor data is completed.

[0044] (Sequence S308) In sequence S308, the multi-stream reception processing unit 203 of the server 200 receives the sensor data, and when the reception of the sensor data is completed, transmits a reception completion notification of the sensor data to the reception application unit 202.

[0045] In the subsequent sequences, after waiting for the waiting time designated in sequence S306, the processes of sequences S305 to S308 are repeated. At this time, for example, the response of API 104 in the first sequence S305 is asynchronous with the response of API 104 in the next sequence S305, and the transmission application unit 102 can transmit data in other streams S without waiting for the completion of data transmission of each stream S.

[0046] <Communication processing when packet loss occurs> Next, with reference to FIG. 4, a case where a packet loss occurs in the communication processing of the communication system 1 will be described. FIG. 4 is a sequence diagram illustrating the communication process of the communication system 1 when a packet loss occurs. In this explanation, we will explain the case where three streams (stream S1, stream S2, and stream S3, in the order they were established) are established and sensor data is transmitted, and packet loss occurs in the second stream (stream S2). However, the same applies when packet loss occurs in the other streams.

[0047] (Steps S305 to S308) Steps S305 to S308 are similar to the communication processing of the communication system 1, and therefore the description thereof will be omitted.

[0048] (Sequence S401: Determination of stream stop threshold) In sequence S305, the transmission application unit 102 executes the API 104 for transmitting sensor data and starts transmission on stream S2. If packet loss occurs, in sequence S401, the multi-stream transmission processing unit 103 retransmits the data in which packet loss occurred on stream S2. However, if retransmission is repeated for a period longer than a certain period due to circumstances such as repeated packet losses, this retransmission process may cause network congestion, resulting in network delays. Furthermore, if data transmission is delayed due to repeated retransmission, there may be cases where the data is unnecessary even if it can be transmitted successfully. For this reason, it is desirable to determine a threshold value for retransmitting data, that is, a stream stop threshold value, and stop a stream whose survival time has been extended. For this reason, the stream control unit 106 determines the stream stop threshold based on, for example, the permissible network delay time of the network requirement input in sequence S302 of FIG.

[0049] (Sequence S402) In sequence S402, the multi-stream transmission processing unit 103 stops the transmission of the stream S2. Specifically, the stream control unit 106 acquires the time since the establishment of the stream S2, and judges whether the time exceeds the stream stop threshold. If the time since the establishment of the stream S2 exceeds the stream stop threshold, the stream control unit 106 discards the sensor data transmitted using the stream S2 and cancels the transmission of the sensor data. If the time does not exceed the stream stop threshold, the stream control unit 106 retransmits the data in which the packet loss occurred.

[0050] The processing of sequence S402 enables efficient use of the line bandwidth of network 300 by canceling the transmission of unnecessary data, even if transmitted late, such as old video data in the case of real-time video transmission.

[0051] <Waiting time calculation process> Next, a calculation process of a waiting time designated in the communication process of the communication system 1 will be described with reference to FIG. FIG. 5 is a flowchart illustrating the waiting time calculation process. 5, the calculation process of the waiting time will be described for a case where at least one stream has been established and transmission of sensor data has started, and then a waiting time is set when another stream is established. However, the calculation and setting of the waiting time can be performed in the same way in other cases, regardless of whether or not there is a preceding stream.

[0052] (Step S501) In step S501, the stream control unit 106 acquires data transmission statistical information such as the number of streams currently being transmitted, the time required for data transmission, etc. Also, information on the waiting time of the latest stream currently being transmitted is acquired.

[0053] (Step S502) In step S502, the stream control unit 106 determines whether the waiting time of the latest stream is a fixed valid waiting time. Here, the fixed waiting time being valid means that the time since the waiting time was fixed is within a predetermined time range. If the fixed waiting time is valid, the process proceeds to step S505, and if it is not valid, the process proceeds to step S505.

[0054] When such a fixed waiting time is effective, using the fixed time again has the advantage of requiring short calculation time since it operates according to a relatively simple algorithm.

[0055] (Step S503) In step S503, the stream control unit 106 compares the data transmission time of the preceding stream with the network delay allowable time, and determines whether or not the data transmission time has exceeded the network delay allowable time. Here, the data transmission time is the time from when the transmission of sensor data using a stream is started, that is, the lifetime of the stream. In addition, the network delay allowable time is a threshold time for determining a delay in the transmission of sensor data, and if the sensor data is transmitted beyond this time, it is considered that a network delay has occurred in the transmission of the sensor data. If the lifetime of the stream is smaller than the network delay allowable time, the process proceeds to step S504, and if it is not valid, the process proceeds to step S506.

[0056] (Step S504) In step S504, if the lifetime of the stream is smaller than the network delay tolerance time, that is, it can be determined that the network has sufficient capacity, the stream control unit 106 performs processing to reduce the waiting time, generate more streams per unit time, and increase the amount of data transmitted. There are various methods for reducing the waiting time, but the stream control unit 106 may reduce the waiting time by a fixed amount. This method makes it possible to increase the throughput in a short time with a small calculation load. Alternatively, the waiting time may be reduced to 1 / N (N is a positive integer equal to or greater than 1). In this case, by exponentially reducing the value of the waiting time, it is possible to suppress an increase in throughput at the beginning of communication and avoid frequent congestion occurrences. This allows the stream control unit 106 to transmit data using more streams.

[0057] (Step S505) In step S505, the stream control unit 106 notifies the transmission application unit 102 of the calculated waiting time, and the calculation process ends.

[0058] (Step S506) If the lifetime of the stream is equal to or greater than the network delay allowable time, in step S506, the stream control unit 106 determines whether or not the waiting time can be fixed. Specifically, the stream control unit 106 analyzes the trend of the data transmission time per stream based on the number of streams used for data transmission and the survival time of the terminated streams, and determines whether the waiting time can be fixed. For example, if the number of streams used for data transmission is increasing but the data transmission time per stream is not decreasing, it is possible that an external factor other than the number of streams is causing a decrease in the available network bandwidth or packet loss. In such a case, it can be determined that the network margin is narrowing, and it is determined that the waiting time cannot be fixed. On the other hand, if the stream lifetime and the network delay tolerance are trending at roughly the same level, it is determined that the waiting time can be fixed, and calculation of the fixed value of the waiting time is proceeded. That is, if the waiting time cannot be fixed, the process proceeds to step S507, and if the waiting time can be fixed, the process proceeds to step S508.

[0059] (Step S507) In step S507, the stream control unit 106 increases the waiting time. At this time, the stream control unit 106 may increase the waiting time by a fixed amount, or may increase the waiting time by M times (M is a positive integer equal to or greater than 1).

[0060] (Step S508) In step S508, the stream control unit 106 calculates a fixed value for the waiting time. The stream control unit 106 may set the waiting time calculated immediately before the data transmission time exceeds the network delay allowable time as a fixed value, for example. Alternatively, the waiting time may be calculated using a fixed value calculation method as shown in FIG. 6, which will be described later.

[0061] <Calculation process for fixed value of waiting time> Next, a process for calculating a fixed value of the waiting time will be described with reference to FIG. FIG. 6 is a flowchart of a process for calculating a fixed value of the waiting time. It is also assumed that the process in FIG. 6 is a process for calculating a fixed value of the waiting time calculated in the process in step S508 in FIG.

[0062] (Step S601) In step S601, the stream control unit 106 extracts information required to calculate an estimated value of the bandwidth of the network 300 from the acquired data transmission statistical information. Information required to calculate an estimate of the bandwidth of network 300 includes, for example, the amount of data transmitted in previously completed data transmissions, the average lifetime per stream during that data transmission, and the average number of streams used in that data transmission.

[0063] (Step S602) In step S602, the stream control unit 106 calculates an estimated value of the bandwidth of the network 300 based on the information extracted from the data transmission statistical information. Specifically, the stream control unit 106 calculates the data transmission time taken to transmit the data based on the following formula.

number

[0064] Next, the stream control unit 106 calculates an estimated value of the bandwidth of the network 300 based on the following formula.

number

[0065] (Step S603) In step S603, the stream control unit 106 calculates the necessary bandwidth per stream to satisfy the network requirements based on the following formula:

number

[0066] (Step S604) In step S604, the stream control unit 106 calculates the number of streams that can be used simultaneously in data transmission. Specifically, the stream control unit 106 calculates the number of streams that can be used simultaneously based on the following formula:

number

[0067] (Step S605) In step S605, the stream control unit 106 calculates the waiting time. Specifically, the waiting time is calculated based on the following formula:

number

[0068] Thus, the process of step S508 in FIG. 5 and the process described in FIG. 6 have a relatively high computational load, but have the advantage that by estimating the available network bandwidth, an appropriate waiting time can be set even if the data size changes.

[0069] <Log information> Next, log information relating to data communication will be described with reference to FIG. FIG. 7 is a diagram showing log information 5000 relating to data communication between the client 100 and the server 200. As shown in FIG. When data is transmitted from the client 100 to the server 200, the communication system 1 outputs a log indicating the state of the data communication. The logs output by communication system 1 are, for example, log information L5001 indicating the state when a session is established, log information L5002 when the waiting time is changed, or log information L5003 when data transmission is stopped. In this embodiment, the stream control unit 106 of the server 200 generates the log information 5000 , but it may be generated by another component, for example, the stream control unit 106 of the client 100 .

[0070] <Actions and Effects> The communication system 1 according to the first embodiment has been described above. The communication system 1 of the present disclosure has a multi-stream transmission processing unit 103 and a multi-stream reception processing unit 203, and the client 100 determines the trend of the transmission time per stream when setting the waiting time for the stream to be transmitted to the server 200. If the transmission time per stream is showing an increasing trend, a process of increasing the waiting time using a predetermined method is performed to set a new waiting time, and if the transmission time per stream is not showing an increasing trend, a process of fixing the waiting time is performed based on data transmission statistical information to newly calculate and set a fixed value. This allows the waiting time to be calculated and the multiplicity of streams to be adjusted based on that waiting time, thereby improving the efficiency of network bandwidth usage, preventing congestion, and enabling HoL blocking to be avoided by transmitting data in multiple streams.

[0071] As explained above, in order to transmit high-quality data in real time from client 100 to server 200, it is necessary to adjust the stream multiplicity and the stream transmission cancellation threshold. However, manual adjustment requires tuning for each network state and device performance, which takes time for development. Therefore, this invention provides a function that automatically adjusts the stream multiplicity and the data transmission cancellation threshold based on requirement information input by the application.

[0072] [Second embodiment] Next, a communication system 2 according to a second embodiment will be described with reference to FIG. FIG. 8 shows an example of a communication system 2. The communication system 2 of the second embodiment differs from the first embodiment in that the transmission application unit 102 includes a sensor data setting management unit 108. Also, the server 200 has a monitoring function unit 206, a planning information management unit 207, and a sensor data setting notification unit 208, is connected to at least two clients 100, and differs from the first embodiment in that the server 200 and the clients 100 have a connection relationship of M (M is a positive integer of 2 or more):1. In the following description, components that are the same as or equivalent to those in the first embodiment described above are given the same reference numerals, and descriptions thereof will be simplified or omitted.

[0073] The communication system 2 is a system that mainly includes at least two or more clients 100, a server 200, and a network 300, and in which the clients 100 transmit data to the server 200 connected via the network 300.

[0074] <Client> The client 100 is a data sender in a communication system 2 that transmits and receives data using a stream. The client 100 mainly includes a sensor unit 101 , a transmission application unit 102 including a sensor data setting management unit 108 , and a multi-stream transmission processing unit 103 .

[0075] <<Sensor data setting management part>> The sensor data setting management unit 108 changes the setting of the sensor data generated by the sensor unit 101 based on the sensor data setting notification received from the server 200 . The sensor data settings include, for example, the resolution of the camera image, the sampling rate for collecting the sensor data, etc. By changing these sensor data settings, the amount of sensor data generated by the sensor unit 101 can be changed.

[0076] <Server> The server 200 is a data receiving destination in the communication system 2 that transmits and receives data using a stream. The server 200 mainly includes a reception application unit 202 , a multi-stream reception processing unit 203 , a display unit 204 , a storage unit 205 , a monitoring function unit 206 , a plan information management unit 207 , and a sensor data setting notification unit 208 .

[0077] <<Monitoring function section>> The monitoring function unit 206 collects requirement information such as the number of connected clients 100, bandwidth usage information of the network 300, and arrival delays of transmitted data as data transmission statistical information for each time series.

[0078] <<Planning Information Management Department>> The plan information management unit 207 manages the plan information of the client 100 connected to the server 200 . The information on the clients 100 managed by the plan information management unit 207 includes, for example, the number of clients 100 connected to the server 200, the area in which the clients operate, and the time period.

[0079] <<Sensor data setting notification section>> The sensor data setting notification unit 208 determines whether or not there is a risk of congestion or data arrival delays occurring in the network 300 based on the data transmission statistical information collected by the monitoring function unit 206 and the planning information managed by the planning information management unit 207. When there is a risk of congestion or a delay in data arrival in the network 300, the sensor data setting notification unit 208 transmits a sensor data setting change notification to the sensor data setting management unit . The sensor data setting management unit 108 changes the setting of the sensor data acquired by the sensor unit 101 based on the setting change notification.

[0080] <Sensor data change notification process> Next, the sensor data change notification process will be described with reference to FIG. FIG. 9 is a sequence diagram relating to the process of changing the settings of the sensor data. There are two patterns for processing notification of changes in sensor data: processing during planning and processing during operation. For example, in the case of a drone, the planning period refers to the time when a flight plan is formulated, and the operation period refers to the time when the drone is flying. First, the case of processing at the time of planning refers to the case where a setting change notification is sent before data is transmitted from the client 100 to the server 200. Moreover, the case where processing is performed during operation refers to the case where a setting change notification is sent from the client 100 to the server 200 while data is being sent. In Fig. 9, the processing during planning is shown in the upper part of Fig. 9 and will be explained in sequences S701 to S703, while the processing during operation is shown in the lower part of Fig. 9 and will be explained in sequences S801 to S803.

[0081] <<When processing during planning>> When processing at the time of planning, the communication system 2 assumes that no data is being transmitted from the client 100 to the server 200.

[0082] (Sequence S701) In sequence S701, the sensor data setting notification unit 208 acquires statistical information on the number of clients 100 transmitting data to the server 200, the operating area, and the network bandwidth usage per client 100, based on the planning information and data transmission statistical information managed by the planning information management unit 207.

[0083] (Sequence S702) In sequence S702, the sensor data setting notification unit 208 determines whether congestion will occur in the network 300 when the client 100 transmits data, based on the number of clients 100 transmitting data to the server 200, the operating area, and statistical information on the network bandwidth used per client 100 obtained in sequence S701. When congestion of the network 300 is predicted, the setting values ​​of the client 100 and the sensor unit 101 are calculated to suppress the amount of sensor data so that the overall amount of data transmitted over the network is at a level that does not cause congestion of the network 300.

[0084] (Sequence S703) In sequence S703, the sensor data setting notification unit 208 transmits the calculated setting value to the transmission application unit 102, and notifies the sensor data setting management unit 108 of a setting change notification. At this time, the client 100 changes the settings of the sensor unit 101 based on the setting change notification received by the sensor data setting management unit 108. In this case, if the change in the setting of the sensor unit 101 is necessary, the sensor unit 101 may be rebooted.

[0085] <<When processing during operation>> When processing during operation, it is assumed that the communication system 2 is transmitting data from the client 100 to the server 200, and the processing sequence from S301 to S308 in FIG. 3 is being repeatedly executed.

[0086] (Sequence S801) In sequence S801, the sensor data setting notification unit 208 acquires data transmission statistical information from the stream control unit 106 in real time. The data transmission statistical information referred to by the sensor data setting notification unit 208 may be obtained by extracting information based on a specific requirement. For example, data transmission statistical information for each operating area may be obtained. In addition, planning information may be reflected in the data transmission statistical information.

[0087] (Sequence S802) In sequence S802, the sensor data setting notification unit 208 determines, based on the data transmission statistical information acquired in sequence S701, whether or not congestion will occur in the network 300 when the client 100 transmits data. If congestion is predicted to occur in the network 300, as in sequence S702, a setting change notification is calculated to reduce the amount of sensor data to be transmitted over the network so that congestion does not occur in the network 300, and if congestion is not predicted, the processing is terminated.

[0088] (Sequence S803) In sequence S803, the sensor data setting notification unit 208 transmits the calculated setting change notification to the transmission application unit 102, and notifies the sensor data setting management unit 108 of the setting change notification. At this time, the client 100 changes the settings of the sensor unit 101 based on the setting change notification received by the sensor data setting management unit 108. In this case, if the change in the setting of the sensor unit 101 is necessary, the sensor unit 101 may be rebooted.

[0089] <Actions and Effects> The communication system 2 according to the second embodiment has been described above. The communication system 2 of the present disclosure has a sensor data setting management unit 108 and a sensor data setting notification unit 208, and based on a setting change notification from the sensor data setting notification unit 208, the sensor data setting management unit 108 can change the settings of the sensor data generated by the sensor unit 101. As a result, in addition to the effects of the first embodiment, when there are multiple clients, by controlling the amount of data to be sent to the network for each client, congestion caused by a lack of bandwidth in the network 300 due to the data volume of the sensor data can be avoided. Furthermore, by controlling the amount of data sent from a client for each operating time period based on the client's operating area and planned information, rather than controlling multiple clients in a uniform manner, it is possible to achieve more precise network operation according to detailed conditions and the transmission and reception of high-quality data.

[0090] As described above, the communication system 2 according to the embodiment of the present disclosure can automatically set the timing of an appropriate transmission request and the transmission stop threshold according to the requirements of the application in communication that avoids HoL blocking by using multiple streams. This makes it possible to improve communication quality while reducing the amount of tuning work required according to the characteristics of the application and network and the performance of the device. Furthermore, even if it is difficult to guarantee communication quality due to network congestion, the amount of data can be reduced according to the situation, and communication quality can be guaranteed by avoiding congestion.

[0091] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the gist of the present invention.

[0092] The present invention can also take the following forms. (Aspect 1) In a communication system in which data is transmitted and received between a server and a client using multiple streams in parallel, The client: If the transmission time per stream is increasing, increase the waiting time and set a new waiting time. If the transmission time per stream is not increasing, a process of fixing the waiting time is carried out based on the data transmission statistical information, and a new fixed value is calculated and set. Communication systems. (Aspect 2) In the communication system according to aspect 1, If the waiting time is fixed and is within a certain period of time since it was fixed, reset the fixed waiting time; If the waiting time is not fixed or if a certain period of time has elapsed since it was fixed, a process of increasing and fixing the waiting time is performed. Communication systems. (Aspect 3) In the communication system according to aspect 1 or 2, The client: When the data transmission time of the preceding stream is smaller than the network delay tolerance time, the waiting time is decreased. Communication systems. (Aspect 4) In the communication system according to any one of aspects 1 to 3, The client: a stream control unit that controls a stop of data transmission of the stream, The stream control unit determining a stream stop threshold for stopping data transmission of the stream that meets a network requirement based on the data transmission statistics; When the transmission time of the data exceeds the stream stop threshold, the data transmission of the stream is stopped. Communication systems. (Aspect 5) In the communication system according to aspect 4, The network requirements are: Application requirements, including FPS (Frames Per Second), are converted and specified as network requirements. Communication systems. (Aspect 6) In the communication system according to any one of aspects 1 to 5, The data transmission statistical information is information including the data transmission time required to transmit the predetermined data, which is calculated based on the number of streams used in the data transmission executed in the past and the transmission time required for the data transmission. Communication systems. (Aspect 7) In the communication system according to any one of aspects 1 to 6, The client: A sensor unit that generates data to be transmitted; A sensor data setting management unit that controls a data volume of the data, The sensor data setting management unit changes the data capacity based on monitoring information. Communication systems. (Aspect 8) In the communication system according to aspect 7, The server receives the data from at least two or more of the clients, The sensor data setting management unit Furthermore, the data capacity is changed based on the planning information. Communication systems. (Aspect 9) In the communication system according to any one of aspects 1 to 8, a first step of increasing the waiting time and setting a new waiting time when the transmission time per stream is increasing; a second step of performing a fixation process for the waiting time based on data transmission statistical information when the transmission time per stream is not increasing, and calculating and setting a new fixed value; A communication method comprising: [Explanation of symbols]

[0093] 100 clients 200 Servers 300 Network 101 Sensor section 102 Transmission Application Part 103 Multi-stream transmission processing unit 203 Multi-stream reception processing unit 104 API 105 Protocol Processing Unit 106 Stream control section 107 Network Requirements Management Department 108 Sensor Data Setting Management Unit 202 Receiving Application Part 204 Display section 205 Storage section 206 Monitoring Function Section 207 Planning Information Management Department 208 Sensor data setting notification unit 5000 Log information< / hdd> < / ram> < / rom> < / cpu> < / api>

Claims

1. In a communication system in which data is transmitted and received between a server and a client using multiple streams in parallel, The client: If the transmission time per stream is increasing, increase the waiting time and set a new waiting time. If the transmission time per stream is not increasing, a process of fixing the waiting time is carried out based on the data transmission statistical information, and a new fixed value is calculated and set. Communication systems.

2. 2. The communication system according to claim 1, If the waiting time is fixed and is within a certain period of time since it was fixed, reset the fixed waiting time; If the waiting time is not fixed or if a certain period of time has elapsed since it was fixed, a process of increasing and fixing the waiting time is performed. Communication systems.

3. 2. The communication system according to claim 1, The client: When the data transmission time of the preceding stream is smaller than the network delay tolerance time, the waiting time is decreased. Communication systems.

4. 2. The communication system according to claim 1, The client: a stream control unit that controls a stop of data transmission of the stream, The stream control unit determining a stream stop threshold for stopping data transmission of the stream that meets a network requirement based on the data transmission statistics; When the transmission time of the data exceeds the stream stop threshold, the data transmission of the stream is stopped. Communication systems.

5. 5. The communication system according to claim 4, The network requirements are: Application requirements, including FPS (Frame Per Second), are converted and defined as the network requirements. Communication systems.

6. 2. The communication system according to claim 1, The data transmission statistical information is information including the data transmission time required to transmit the predetermined data, which is calculated based on the number of streams used in the data transmission executed in the past and the transmission time required for the data transmission. Communication systems.

7. 2. The communication system according to claim 1, The client: A sensor unit that generates data to be transmitted; A sensor data setting management unit that controls a data volume of the data, The sensor data setting management unit changes the data capacity based on monitoring information. Communication systems.

8. 8. The communication system according to claim 7, The server receives the data from at least two or more of the clients, The sensor data setting management unit Furthermore, the data capacity is changed based on the planning information. Communication systems.

9. 2. The communication system according to claim 1, a first step of increasing the waiting time and setting a new waiting time when the transmission time per stream is increasing; a second step of performing a fixation process for the waiting time based on data transmission statistical information when the transmission time per stream is not increasing, and calculating and setting a new fixed value; A communication method comprising:

Images