Communication control method
The communication control method dynamically switches between P2P and client-server methods based on ensemble state and load conditions to optimize network performance and minimize delays, addressing the challenges of communication delays in ensemble performances.
Patent Information
- Application Number
- JP2024161169
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2040-09-01
AI Technical Summary
Existing communication methods for ensemble performances over networks suffer from communication delays, which are exacerbated by peer-to-peer (P2P) type communication's increased processing load and reduced by client-server type communication's delays, necessitating a dynamic switching method to optimize performance.
A communication control method that switches between P2P and client-server communication methods based on the ensemble state and load conditions, using a communication management server to determine the optimal method for each data type to minimize delays and loads.
This approach effectively reduces communication delays during ensemble performances by dynamically adjusting communication methods, ensuring low latency and load management, thereby enhancing the overall performance and synchronization of data transmission.
Smart Images

Figure 2024166365000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for controlling communication. [Background technology]
[0002] A technology has been developed that enables musical ensemble performances even in remote locations by connecting multiple communication points where musical instruments are played via a network. Communication delays caused by network connections make it difficult to play ensembles. Therefore, it is desirable to create an environment that reduces communication delays, but communication delays cannot be eliminated. For this reason, a technology for reducing the effects of delays assuming that communication delays exist is disclosed in, for example, Patent Document 1. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2005-195982 A Summary of the Invention [Problem to be solved by the invention]
[0004] Methods of communicating in real time between multiple communication points (multiple communication terminals) include, for example, Peer to Peer communication (hereinafter referred to as P2P communication) and client-server communication. P2P communication is a method in which communication terminals at each communication point connect on an equal footing. Client-server communication is a method in which communication terminals at each communication point connect via a server. Due to the difference in these communication methods, P2P communication can reduce communication delays, but the processing load and communication volume of the communication terminals is likely to increase. On the other hand, client-server communication can suppress increases in the load and communication volume of the communication terminals, but communication delays become large. Such a server has functions such as a Selective Forwarding Unit (SFU) and a Multipoint Control Unit (MCU).
[0005] In the above-mentioned ensemble, it is necessary to minimize communication delays. For this reason, it is desirable to use P2P communication as the communication method when performing an ensemble. However, if the processing load on the communication terminal becomes large, the processing load may result in large delays or affect other processing on the communication terminal. On the other hand, depending on the situation at each communication point, it may not necessarily be necessary to use P2P communication.
[0006] One object of the present invention is to switch communication methods depending on the situation between a plurality of communication points. [Means for solving the problem]
[0007] According to one embodiment of the present invention, there is provided a communication control method, which includes obtaining a comparison result between a first feature in first streaming data transmitted from a first terminal by a first communication method and a second feature in second streaming data transmitted from a second terminal by the first communication method, and, if the first feature and the second feature have a first relationship based on the comparison result, transmitting a first switching signal to the first terminal and the second terminal for switching streaming data that is subject to communication control and that is transmitted between the first terminal and the second terminal by the first communication method from transmission by the first communication method to transmission by a second communication method different from the first communication method. Effect of the Invention
[0008] According to the present invention, the communication method can be switched depending on the situation between a plurality of communication points. [Brief description of the drawings]
[0009] [Figure 1] 1 is a diagram illustrating a configuration of a communication system according to an embodiment of the present invention. [Diagram 2] FIG. 4 is a diagram illustrating a communication control table in one embodiment of the present invention. [Diagram 3]4 is a flowchart illustrating a communication switching method of a communication terminal in one embodiment of the present invention. [Figure 4] 11 is a flowchart illustrating a communication control method of a management server in one embodiment of the present invention. [Diagram 5] 11A and 11B are diagrams illustrating a method for detecting an ensemble state in one embodiment of the present invention. [Figure 6] 11A and 11B are diagrams illustrating a method for detecting an ensemble state in one embodiment of the present invention. [Figure 7] FIG. 4 is a diagram illustrating a method for controlling a communication mode in one embodiment of the present invention. [Figure 8] FIG. 2 is a diagram illustrating the flow of data between communication points. [Figure 9] FIG. 2 is a diagram illustrating a method for synchronizing each piece of data. [Figure 10] 11 is a diagram for explaining a method of synchronizing each piece of data when video data Dv is not synchronized. FIG. [Figure 11] FIG. 11 is a diagram for explaining a method of synchronizing each piece of data in a specific mode. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] A communication system according to an embodiment of the present invention will be described in detail below with reference to the drawings. The embodiment described below is an example of an embodiment of the present invention, and the present invention is not limited to these embodiments. In the drawings referred to in this embodiment, the same parts or parts having similar functions are given the same or similar symbols (symbols with A, B, etc. added after the numbers), and repeated explanations may be omitted.
[0011] [1. Communication system configuration] FIG. 1 is a diagram for explaining the configuration of a communication system in one embodiment of the present invention. The communication system includes a communication management server 1 connected to a network NW such as the Internet. The communication management server 1 controls communication between a plurality of communication bases connected to the network NW according to the status of each communication base. The functions of the communication management server 1 described later may be realized by a plurality of servers working together. In this example, two communication methods are used for communication between the communication bases. The two communication methods are P2P type communication and client-server type communication. In FIG. 1, three communication bases T1, T2, and T3 are illustrated, but the number is not limited to this number. When the communication bases T1, T2, and T3 are not distinguished from each other, they are simply referred to as communication bases. The SFU server 80 is a server used in client-server type communication and mediates between each communication base.
[0012] A communication terminal 20 is disposed at each communication base. An electronic musical instrument 30, a sound collection device 40, an imaging device 50, a sound emission device 60, and a display device 70 are connected to the communication terminal 20. Although a communication terminal 20 is always present at each communication base, at least one of the electronic musical instrument 30, the sound collection device 40, the imaging device 50, the sound emission device 60, and the display device 70 does not have to be connected to the communication terminal 20. At least one of the electronic musical instrument 30, the sound collection device 40, the imaging device 50, the sound emission device 60, and the display device 70 may be configured as an integrated device with the communication terminal 20.
[0013] The electronic musical instrument 30 includes performance operators, a sound source that outputs sound data Da in response to operations on the performance operators, and a data output unit that outputs sound generation control data Dc corresponding to the operations. In this example, the electronic musical instrument 30 is an electronic piano having keys as performance operators. The sound data Da and sound generation control data Dc are output to the communication terminal 20. The sound data Da is data representing a sound waveform signal, and may be output to the sound emission device 60. The sound generation control data Dc is, for example, M This is data for controlling a sound source in accordance with a predetermined standard such as the IDI standard. In this case, the sound generation control data Dc includes information that specifies the sound generation content, such as information on the sound generation start timing (note on) and information on the sound generation end timing (note off).
[0014] The sound source can also output sound data Da under the control of the sound generation control data Dc. The electronic musical instrument 30 may include a drive unit such as a solenoid that drives the performance controls in response to the sound generation control data Dc input from the outside. When the sound generation control data Dc is received from another communication point, the sound generation control data Dc may be supplied to such a sound source or drive unit. Driving the performance controls creates a situation equivalent to a performance being performed on the electronic musical instrument 30, and sound data Da is generated in response to the sound generation control data Dc.
[0015] The sound collection device 40 has, for example, a microphone or an input terminal for a sound waveform signal, and outputs the sound input to the microphone or the sound waveform signal input to the input terminal to the communication terminal 20 as sound data Da.
[0016] The imaging device 50 has, for example, a camera, and outputs video data Dv corresponding to an image captured by the camera to the communication terminal 20. In the following description, the image includes both a still image and a video image.
[0017] The sound emitting device 60 has, for example, a speaker, and outputs from the speaker a sound indicated by the sound data Da supplied from the communication terminal 20. The sound data Da supplied to the sound emitting device 60 is sound data Da transmitted from another communication base, but may also include sound data Da generated at the sound emitting device 60's own communication base.
[0018] The display device 70 has, for example, a liquid crystal display, and displays on the liquid crystal display an image indicated by the video data Dv supplied from the communication terminal 20. The video data Dv supplied to the display device 70 is video data Dv transmitted from another communication base, but may also include video data Dv generated at the display device 70's own communication base.
[0019] [2. Configuration of communication terminal] The communication terminal 20 includes a control unit 21, a storage unit 23, a communication unit 25, and an operation unit 27. The control unit 21 includes a CPU, a RAM, a ROM, etc. The control unit 21 executes a program stored in the storage unit 23 by the CPU, thereby performing processing according to instructions defined in the program. In this example, a program for performing processing to realize a method for switching communication (a communication switching method) is stored in the storage unit 23. The communication switching method includes processing for switching a communication method for transmitting data based on an instruction from the communication management server 1.
[0020] The communication unit 25 includes a communication module, and is connected to the network NW to transmit and receive various data to and from external devices such as the communication management server 1, the communication terminals 20 at other communication bases, and the SFU server 80. Data transmitted between the communication unit 25 (communication terminal 20) and external devices other than the communication management server 1 includes sound generation control data Dc, sound data Da, and video data Dv, and is transmitted as streaming data. The communication method used for transmitting this data is set under the control of the control unit 21. As described above, the communication method in this example includes P2P communication and client-server communication.
[0021] The operation unit 27 includes operation devices such as a mouse and a keyboard, receives operations performed by the user on the operation devices, and outputs a signal corresponding to the operation to the control unit 21.
[0022] The storage unit 23 includes a storage device such as a non-volatile memory, and is executed by the control unit 21. The storage unit 23 stores a program. In addition, various data used in the communication terminal 20 is stored. The data stored in the storage unit 23 includes a communication control table. This program may be provided to the communication terminal 20 in a state where it is stored in a computer-readable recording medium such as a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, or a semiconductor memory, as long as it is executable by a computer. In this case, the communication terminal 20 may be provided with a device for reading the recording medium. Also, this program may be provided by downloading it via the communication unit 25.
[0023] 2 is a diagram illustrating a communication control table in one embodiment of the present invention. The communication control table specifies the communication method used to transmit the sound generation control data Dc, the sound data Da, and the video data Dv in each of a plurality of communication modes. The plurality of communication modes include modes A, B, C, and D. The communication methods include client-server communication (SFU) and P2P communication (P2P).
[0024] In mode A, all data is transmitted using client-server type communication (SFU). In mode D, all data is transmitted using P2P type communication (P2P). On the other hand, in mode B, the sound generation control data Dc is transmitted using P2P type communication (P2P), and the sound data Da and video data Dv are transmitted using client-server type communication (SFU). In mode C, the sound generation control data Dc and sound data Da are transmitted using P2P type communication (P2P), and the video data Dv is transmitted using client-server type communication (SFU).
[0025] When the communication terminal 20 is instructed by the communication management server 1 to set a communication mode, the control unit 21 refers to this communication control table to determine the communication method for transmitting each piece of data corresponding to that communication mode. In this way, the control unit 21 sets the communication method corresponding to the instructed communication mode in the communication unit 25. The setting of the communication mode by the communication management server 1 will be described later.
[0026] [3. Communication switching method] Next, a description will be given of a communication switching method in the communication terminal 20. As described above, each process in the communication switching method is executed by the control unit 21. The process of the communication switching method described below is started by, for example, a user instruction input to the operation unit 27.
[0027] 3 is a flowchart for explaining a communication switching method of a communication terminal in one embodiment of the present invention. The communication terminal 20 connects to the communication management server 1 (step S110). At this time, the communication terminal 20 specifies a session room in which the communication base to be connected will participate. A new session room can also be opened in the communication management server 1. The communication terminals 20 at multiple communication bases participating in this session room are the terminals that will transmit data to each other.
[0028] The communication terminal 20 starts transmitting the feature information, communication load information, and processing load information to the communication management server 1 (step S120). During the period in which the communication terminal 20 is connected to the communication management server 1, the communication terminal 20 transmits the feature information, communication load information, and processing load information to the communication management server 1 at predetermined time intervals (e.g., every 100 milliseconds).
[0029] The feature information includes the feature of the sound generation control data Dc and the sound data Da transmitted as streaming data. In this example, the feature is information indicating whether or not sound is being generated at the time of transmission. Therefore, the feature information on the sound data Da indicates the sound generation period of the sound data Da by indicating that sound is generated during the period when the volume level obtained from the sound data Da exceeds a predetermined value, and sound is not generated during other periods. The feature information on the sound generation control data Dc indicates the period from the note-on indicated by the sound generation control data Dc to the note-on indicated by the sound generation control data Dc. The sound generation control data Dc indicates the sound generation period by indicating that sound is generated during the period from the sound generation control data Dc to the time when the tone is turned off and that sound is not generated during the other periods. The feature amount information regarding the sound generation control data Dc may be configured to indicate that sound is generated when the sound generation control data Dc is converted into sound data Da by a sound source and the volume level obtained from the sound data Da exceeds a predetermined value.
[0030] The communication load information is information indicating the load of the communication line used when the communication terminal 20 transmits data. The communication load is obtained, for example, by measuring the round trip time in P2P communication between the communication terminal 20 and another communication terminal 20. In general, the communication load is greater in P2P communication than in client-server communication because the amount of data transmitted and received is greater. Therefore, if there is no capacity in the communication line, low latency cannot be achieved even in P2P communication.
[0031] The processing load information is information indicating the processing load in the communication terminal 20. The processing load corresponds to, for example, the load of a CPU. In general, the amount of data transmitted and received is greater in P2P communication than in client-server communication, and therefore the processing load is greater due to encoding and decoding, etc. Therefore, if the communication terminal 20 does not have sufficient data processing capacity, low latency cannot be achieved even in P2P communication. This processing load may also include that caused by other programs executed in parallel in the communication terminal 20.
[0032] When the communication terminal 20 connects to the communication management server 1, it receives an instruction to set the communication mode from the communication management server 1 and sets the communication mode (step S130). The communication mode specified by the communication management server 1 is the communication mode set in the session room specified by the communication terminal 20. If a new session room is opened, for example, and no communication mode is set in this session room, the communication management server 1 instructs the setting of communication mode A. If the session room is already opened, another communication mode (mode B, mode C, or mode D) may be set by a communication control method described later.
[0033] The communication terminal 20 refers to the communication control table and transmits each data according to the set communication mode. Then, the communication terminal 20 waits until it receives a switching signal S(X) from the communication management server 1 (step S140; No). Here, X indicates one of A, B, C, and D. A, B, C, and D correspond to mode A, mode B, mode C, and mode D, respectively. When it receives the switching signal S(X) (step S140; Yes), the communication terminal 20 switches the communication mode by setting the communication mode X according to the switching signal S(X) (step S150). Then, the communication terminal 20 waits until it receives the switching signal S(X) (step S140; No).
[0034] The communication terminal 20 is controlled by the communication management server 1 so as to change the communication method used for transmitting each of the sound generation control data Dc, the sound data Da, and the video data Dv.
[0035] [4. Configuring the communication management server] The communication management server 1 includes a control unit 11, a storage unit 13, and a communication unit 15. The control unit 11 includes a CPU, RAM, ROM, etc. The control unit 11 executes a program stored in the storage unit 13 by the CPU, thereby performing processing according to instructions defined in the program. This program includes a program for performing processing to realize a method for controlling communication of the communication terminal 20 (a communication control method). The communication control method includes processing for instructing the communication terminal 20 to switch the communication method for transmitting data. The details of this processing will be described later. The communication unit 15 includes a communication module, and is connected to the network NW to transmit various data to and from the communication terminals 20 at each communication point.
[0036] The storage unit 13 includes a storage device such as a hard disk, and stores a program executed by the control unit 11. In addition, various data used in the communication management server 1 is stored. This program only needs to be executable by a computer, and may be provided to the communication management server 1 in a state stored in a computer-readable recording medium such as a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, or a semiconductor memory. In this case, the communication management server 1 only needs to be equipped with a device for reading the recording medium. In addition, this program may be provided by downloading it via the communication unit 15.
[0037] [5. Communication control method] Next, a communication control method in the communication management server 1 will be described. As described above, each process in the communication control method is executed by the control unit 11. The process of the communication control method described below is started, for example, when multiple communication locations participate in a session room. At this time, the performance mode is set to mode A. Therefore, the transmission of sound generation control data Dc, sound data Da, and video data Dv between each communication location is all set as client-server type communication.
[0038] 4 is a flowchart illustrating a communication control method of the management server in one embodiment of the present invention. The communication management server 1 compares feature information received from the communication terminals 20 at a plurality of communication points, and waits until an ensemble state is detected (step S210; No).
[0039] The ensemble state refers to a state in which a performance is being performed at a plurality of communication locations. The performance at each communication location may be, for example, a performance using an electronic musical instrument 30 or a performance using an acoustic musical instrument. When the performance is using an electronic musical instrument 30, at least one of the sound generation control data Dc and the sound data Da is supplied to the communication terminal 20. When the performance is using an acoustic musical instrument, the sound generated by the acoustic musical instrument is input to the sound collection device 40, and the sound data Da is supplied to the communication terminal 20.
[0040] As described above, the feature information transmitted from the communication terminal 20 of each communication point includes information indicating whether the sound generation control data Dc and sound data Da transmitted as streaming data are being generated at each time. Therefore, by comparing this feature information along the time axis, it is possible to detect whether sounds are being generated simultaneously at multiple communication points. However, even if it is possible to detect the occurrence of a sound based on the feature information, it is possible that the sound is not necessarily generated as a result of playing. The method of detecting the ensemble state in this example is a method of excluding sounds not generated as a result of playing as much as possible.
[0041] 5 and 6 are diagrams for explaining a method for detecting an ensemble state in one embodiment of the present invention. These diagrams show the sounding period (mesh part in the diagram) at each communication point along time t, based on feature information transmitted from the communication terminals 20 at the communication points T1, T2, and T3. The feature indicates the sounding of the sounding control data Dc at the communication point T1, and the sounding of the sound data Da at the communication points T2 and T3. FIG. 6 corresponds to the situation after a short time has passed since the situation in FIG. 5.
[0042] A specific example of the ensemble detection method will be described. Five intervals W1 to W5 are defined by dividing the period from 5 seconds before the ensemble detection determination time DP (current time) to the determination time DP into 1-second intervals. The communication management server 1 determines whether there is a sound generation period at multiple communication points in each interval (determination period).
[0043] In an actual ensemble, the sounding periods do not overlap, that is, there are cases where there are no simultaneous sounds. Therefore, even if the sounding periods at multiple communication points do not overlap, the sounding periods in one section may be If there exists an ensemble sound in the section, the section is determined as a section in which an ensemble sound has occurred. The "multiple communication bases" referred to here may be different from all communication bases participating in the session room, that is, it refers to at least two of the communication bases participating in the session room. When the communication management server 1 determines that an ensemble sound has occurred in the sections W1 to W5, it detects that the session room in which these communication bases participate has entered an ensemble state.
[0044] The time going back from the determination time DP (the total time of the sections W1 to W5), the number of divided sections, and the time of each section are merely examples and may be set to various values. The lengths of the sections W1 to W5 may be different. In this case, the sections further away from the determination time DP may be shorter, or the sections closer to the determination time DP may be shorter. The sections W1 to W5 do not have to be consecutive. In other words, adjacent sections may be spaced apart in time.
[0045] In Fig. 5, since only communication point T1 produces sound in section W1, it is determined that there is no ensemble sound in section W1. On the other hand, it is determined that there is an ensemble sound in sections W2 to W5. As described above, it is also determined that there is an ensemble sound in section W2, where the sound production period of communication point T1 and the sound production period of communication point T2 do not overlap. At this point in time, since it has been determined that there is no ensemble sound in section W1, it has not been detected that the session rooms in which communication points T1 to T3 participate have entered an ensemble state.
[0046] 6, it is determined that there is an ensemble sound in the sections W1 to W5. Therefore, the communication management server 1 detects that the session room in which the communication points T1 to T3 participate has entered an ensemble state. When the ensemble state is detected, even if the above condition is no longer satisfied, the session room is maintained in the ensemble state until a non-ensemble state, which will be described later, is detected.
[0047] Returning to Fig. 4, the explanation will be continued. When an ensemble state is detected (step S210; Yes), the communication management server 1 transmits a switching signal S(D) to the communication terminal 20 at each communication point (step S230). When the switching signal S(D) is received by the communication terminal 20 as described above, the communication mode is switched from mode A to mode D. That is, the transmission of the sound generation control data Dc, the sound data Da, and the video data Dv between each communication point is all switched from client-server type communication to P2P type communication.
[0048] The communication management server 1 compares the feature information received from the communication terminals 20 at the multiple communication points, and performs the communication mode switching process until a non-ensemble state is detected (step S310; No, step S400). A non-ensemble state refers to a state in which no performance is being performed at any communication point, or a state in which performance is being performed at only one communication point.
[0049] A specific example of a method for detecting a non-ensemble state is the opposite of the method for detecting an ensemble state. That is, when it is determined that there is no ensemble sound in any of the sections W1 to W5 corresponding to the determination time point DP, the non-ensemble state is detected.
[0050] When a non-ensemble state is detected (step S310; Yes), the communication management server 1 transmits a switching signal S(A) to the communication terminal 20 of each communication point (step S900). As described above, when the switching signal S(A) is received by the communication terminal 20, the communication mode is set to mode A. That is, the transmission of the sound generation control data Dc, the sound data Da, and the video data Dv between each communication point is all set as client-server type communication. Then, the communication management server 1 waits until an ensemble state is detected again (step S210; No).
[0051] The communication mode switching process (step S400) is a process of switching the communication mode in accordance with a load value Ld. The load value Ld is determined based on communication load information and processing load information transmitted from each communication terminal 20. It is an index of the magnitude of the load calculated by the communication management server 1 based on the load information. In this example, the communication load and processing load corresponding to the multiple communication terminals 20 each range from 0% to 100%, and the maximum value is the load value Ld.
[0052] Fig. 7 is a diagram for explaining a method of controlling a communication mode in one embodiment of the present invention, showing conditions when the communication mode is switched in response to a load value Ld in the communication mode switching process.
[0053] For example, when the communication mode is mode D and the load value Ld becomes larger than the threshold value TC1, the communication management server 1 transmits a switching signal S(C) to each communication terminal 20 to switch the communication mode to mode C. When the communication mode is mode C and the load value Ld becomes larger than the threshold value TB1, the communication management server 1 transmits a switching signal S(B) to each communication terminal 20 to switch the communication mode to mode B. On the other hand, when the communication mode is mode C and the load value Ld becomes smaller than the threshold value TC21, the communication management server 1 transmits a switching signal S(D) to each communication terminal 20 to switch the communication mode to mode D.
[0054] Here, TC1>TC2. By doing so, even if Ld becomes smaller than TC1 immediately after Ld becomes larger than TC1 and the communication mode is switched to mode C, the communication mode is not immediately switched to mode D. Similarly, the relationships of the other thresholds are TB1>TB2 and TA1>TA2. The load of mode A is less than the load of mode B, which is less than the load of mode C, which is less than the load of mode D. Therefore, TA1, TB1 and TC1 may be set as the same threshold. TA2=TB2=TC2 may be set as the same threshold.
[0055] In this manner, in the communication mode switching process (step S400), when the load value Ld increases, the communication management server 1 changes the communication mode to a mode with a lighter load than the current mode, and when the load value Ld decreases, the communication mode to a mode with a smaller communication delay than the current mode. This communication mode switching process continues until the non-ensemble state described above is detected.
[0056] [6. Example of data transmission and reception] Next, an example of a communication method applied to the transmission of sound generation control data Dc, sound data Da, and video data Dv between communication points using the above-mentioned communication control method and communication switching method will be described.
[0057] Fig. 8 is a diagram explaining the flow of data between communication points. When no performance is taking place at each point, the communication mode remains set to mode A. In this case, all of the sound generation control data Dc, sound data Da, and video data Dv are transmitted and received between the communication points via the SFU server 80 (paths shown by dashed lines in Fig. 8). That is, client-server type communication is applied to all data. In this situation, even if the communication delay is large, no particular problem occurs because no ensemble is taking place, so reducing the communication load and processing load is prioritized.
[0058] Assume that a performance is being performed at communication point T1. In this state, the communication mode remains in mode A. If a performance is further performed in communication mode T2, an ensemble state is detected. This causes the communication mode to switch to mode D. Therefore, the sound generation control data Dc, sound data Da, and video data Dv are all transmitted and received directly between communication points via P2P communication (paths shown by solid lines in FIG. 8).
[0059] After that, if the load value Ld increases, the communication mode is switched to mode C. Therefore, the sound generation control data Dc and the sound data Da are maintained in transmission by P2P type communication, but only the video data Dv is switched to transmission via the SFU server 80. When the load value Ld changes, the communication mode is switched in accordance with the change, and the communication method for each data is switched.
[0060] As data that reduces communication delays, video data Dv has the lowest priority, and sound generation control data Dc has the highest priority. Even if the video data Dv is delayed, so long as the sound delay is small, it will not cause a significant hindrance to playing together. Therefore, when the communication mode is switched from mode D to mode C, the transmission of the video data Dv is switched to client-server type communication. Since the amount of data of the sound generation control data Dc is small, maintaining P2P type communication is unlikely to increase the load. Therefore, in communication modes other than mode A, the transmission of the sound generation control data Dc uses P2P type communication.
[0061] When the performance is interrupted at at least one of the communication points T1 and T2, a non-ensemble state is detected. As a result, the communication mode is switched to mode A, and all of the sound generation control data Dc, the sound data Da, and the video data Dv are directly transmitted between the communication points in a P2P type communication.
[0062] [7. Synchronization between each data] Next, a method for synchronizing the sound output from the sound emitting device 60 with the video displayed on the display device 70 at a communication point that receives the sound generation control data Dc, the sound data Da, and the video data Dv will be described. Synchronization here refers to maintaining the temporal relationship of each part of the data at the data transmitting side at the data receiving side as well. The following description assumes that the data transmitting side is communication point T1 and the data receiving side is communication point T2.
[0063] Fig. 9 is a diagram explaining a method for synchronizing each piece of data. In Fig. 9, time tp indicates the timing at which a specific part of each piece of data is transmitted from communication point T1, and time te indicates the timing at which data received at communication point T2 is synchronized and output.
[0064] Time dc is a time equivalent to a communication delay of the sound generation control data Dc. Therefore, communication point T2 receives the sound generation control data Dc after time dc has elapsed from time tp. Time da is a time equivalent to a communication delay of the sound data Da. Therefore, communication point T2 receives the sound data Da after time da has elapsed from time tp. Time dv is a time equivalent to a communication delay of the video data Dv. Therefore, communication point T2 receives the video data Dv after time dv has elapsed from time tp.
[0065] In this example, in order to synchronize with the video data Dv that has the greatest communication delay, the time te corresponds to the reception time of the video data Dv. Therefore, at the communication point T2, the communication terminal 20 supplies the sound generation control data Dc to the electronic musical instrument 30 after the passage of time htc (=time dv-time dc) since receiving the sound generation control data Dc. The sound generation control data Dc is buffered in the communication terminal 20 until the passage of time htc. The electronic musical instrument 30 converts the sound generation control data Dc into sound data Da using a sound source and supplies it to the sound emitting device 60. As a result, a sound corresponding to the sound generation control data Dc is output from the sound emitting device 60.
[0066] At communication point T2, the communication terminal 20 supplies the sound data Da to the sound emitting device 60 after time hta (=time dv-time da) has elapsed since receiving the sound data Da. The sound data Da is buffered in the communication terminal 20 until time hta has elapsed. As a result, sound corresponding to the sound data Da is output from the sound emitting device 60. At communication point T2, when the communication terminal 20 receives the video data Dv, it supplies the video data Dv to the display device 70. As a result, an image corresponding to the video data Dv is displayed on the display device 70. In this manner, Furthermore, the sound corresponding to the sound generation control data Dc, the sound corresponding to the sound data Da, and the image corresponding to the video data Dv are synchronized.
[0067] The times htc and hta may be obtained in advance using known technology in relation to each communication point (the relationship from communication point T1 to communication point T2 in the example of Figure 9), or may be obtained from the time difference between timestamps obtained sequentially by assigning a timestamp corresponding to the transmission time to each data.
[0068] The time htc and the time hta may be obtained by a method other than the above. For example, at the communication point T1, a specific action by a performer is imaged, and a specific sound is recorded simultaneously with the specific action. The sound data Da and video data Dv obtained in this way may be received at the communication point T2, and the time hta may be obtained based on the difference between the specific action and the specific sound. At the communication point T1, the sound generation control data Dc and the sound data Da are obtained by playing the electronic musical instrument 30. The sound generation control data Dc and the sound data Da may be received at the communication point T2, and the time htc may be obtained based on the difference between the note-on included in the sound generation control data Dc and the sound generation start timing of the sound data Da (corresponding to the time htx shown in FIG. 10) and the time hta.
[0069] When the video data Dv is significantly delayed, particularly when the communication mode is mode C, the video data Dv may be excluded from synchronization.
[0070] FIG. 10 is a diagram for explaining a method of synchronizing each data when the video data Dv is not synchronized. Unlike the example shown in FIG. 9, the communication base T2 receives the video data Dv transmitted at time tp after time te. In this example, the time te corresponds to the reception time of the sound data Da in order to synchronize with the sound data Da, which has the largest communication delay among the data to be synchronized. Therefore, at the communication base T2, the communication terminal 20 supplies the sound generation control data Dc to the electronic musical instrument 30 after time htx (=time da-time dc) has elapsed since receiving the sound generation control data Dc. The electronic musical instrument 30 converts the sound generation control data Dc into sound data Da by a sound source and supplies the sound generation device 60. As a result, a sound corresponding to the sound generation control data Dc is output from the sound generation device 60.
[0071] At the communication point T2, when the communication terminal 20 receives the sound data Da, it supplies the sound data Da to the sound emitting device 60. As a result, a sound corresponding to the sound data Da is output from the sound emitting device 60. In this way, the sound corresponding to the sound control data Dc and the sound corresponding to the sound data Da are synchronized. On the other hand, the video data Dv is not synchronized with these data.
[0072] Next, it is assumed that the electronic musical instrument 30 at the communication point T2 has a drive unit that drives the performance operators based on the sound generation control data Dc. In this case, a drive time is required from when the sound generation control data Dc is supplied to the electronic musical instrument 30 until the performance operators are driven. Each data may be synchronized taking into account the drive time. In such a synchronization, a specific mode is set.
[0073] Fig. 11 is a diagram for explaining a method of synchronizing each piece of data in a specific mode. The time dm shown in Fig. 11 corresponds to the drive time required from when the sound generation control data Dc is supplied until when the performance operators are driven. The time dm may be preset as a characteristic of the drive unit of the electronic musical instrument 30, or may be measured in advance.
[0074] In this example, the time until the sound data Da is output from the electronic musical instrument 30 in response to the sound generation control data Dc is the longest. Therefore, the time te corresponds to the timing at which the sound data Da is output from the electronic musical instrument 30 whose performance operators are driven in response to the sound generation control data Dc. Therefore, at the communication point T2, the communication terminal 20 supplies the sound data Da to the sound emitting device 60 after the time hty (=time dc+time dm-time da) has elapsed since receiving the sound data Da. As a result, a sound corresponding to the sound data Da is output from the sound emitting device 60.
[0075] At the communication point T2, the communication terminal 20 supplies the video data Dv to the display device 70 after a time htv (=time dc+time dm-time dv) has elapsed since receiving the video data Dv. As a result, an image corresponding to the video data Dv is displayed on the display device 70. In this way, the sound corresponding to the sound control data Dc, the sound corresponding to the sound data Da, and the image corresponding to the video data Dv are synchronized.
[0076] When the time until the sound data Da is output from the electronic musical instrument 30 in response to the sound generation control data Dc is the longest, that is, when the time dm is long, the sound data Da and the video data Dv do not need to be transmitted by P2P type communication depending on the time da and the time dv. For example, even if the sound data Da and the video data Dv are transmitted from the communication point T1 by client-server type communication, if they are received at the communication point T2 before the time te, the sound data Da and the video data Dv do not need to be transmitted by P2P type communication. Therefore, in this case, the communication mode may be set to mode C or D or mode B.
[0077] As described above, in the communication system of one embodiment, the communication method can be switched depending on the situation between the multiple communication bases. Therefore, it is possible to control communication by applying a communication method that reduces communication delay when the performances at the multiple communication bases are in an ensemble state, and applying a communication method that reduces the communication load and processing load when the performances are not in an ensemble state. When the communication load or processing load becomes large, the communication method can be controlled for each type of data.
[0078] <Modification> Although one embodiment of the present invention has been described above, this embodiment can be modified in various ways as described below. Furthermore, the above-described embodiment and the modifications described below can be applied in combination with each other.
[0079] (1) The communication management server 1 may have the functions of the SFU server 80. In this case, the communication management server 1 may detect the ensemble state by using data transmitted by client-server communication instead of feature information.
[0080] (2) The switching signal transmitted from the communication management server 1 to the communication terminal 20 is a signal for switching the performance mode, but instead of specifying the performance mode after switching, it may be a signal for specifying the communication method of each data. In this case, it may be in a form for specifying data for which the communication method needs to be changed. In this way, the communication terminal 20 does not need to have a communication control table.
[0081] (3) The timing at which P2P-type communication connecting each communication point is established may be the timing at which P2P-type communication becomes necessary depending on the communication mode, or may be the timing before the start of detection of the ensemble state. When there is no data to be transmitted by P2P-type communication, the establishment of P2P-type communication may be terminated.
[0082] (4) When the communication method is switched, the communication terminal 20 may output data that has been received in the communication method before the switch and that has been buffered in the communication terminal 20 to the electronic musical instrument 30, the sound output device 60, and the display device 70 until the data is received in the communication method after the switch.
[0083] (5) In the embodiment, the communication points T1 and T2 are performing together, so that the session For the communication point T3 that was not compared when the room was detected as being in an ensemble state, a process different from the set communication mode may be executed. For example, since it is not necessary to reduce communication delays for data transmitted from the communication point T3, client-server type communication may be used.
[0084] (6) The communication modes may be only modes A and D. In this case, there is no need to switch the communication mode depending on the communication load and processing load.
[0085] (7) The moving image data Dv may be transmitted by client-server communication in all communication modes and may be excluded from streaming data that is subject to control of the communication method.
[0086] (8) When the communication load and the processing load exceed a predetermined value, the communication terminal 20 may execute a process to reduce these loads. The communication terminal 20 may reduce the load by, for example, a process to reduce the video size, a process to lower the video resolution, a process to lower the video frame frequency, a process to lower the sound sampling frequency, a process to increase the compression rate, and a process to reduce the audio channel (a process to change stereo to monaural). The adjustment range of these processes is predetermined. When the process to reduce the load is executed in this way, the above-mentioned load value Ld does not increase until adjustment within the adjustment range is no longer possible.
[0087] (9) The ensemble state may be detected by comparing the video data Dv. In this case, the feature information includes a feature related to the video data Dv. This feature may be, for example, information indicating whether or not an instrument is included in the image as a result of analyzing an image shown by the video data Dv, or information indicating whether or not a moving person is included in the image. The ensemble state may be detected by using a period during which an instrument is included in the image or a period during which a moving person is included in the image, instead of the sounding period in the embodiment. In this way, if the communication method between communication bases is controlled according to the status of each communication base, various methods can be applied to detect the status of each communication base.
[0088] Even in this case, the video data Dv may be excluded from the streaming data to be controlled by the communication method as in the above modification (7). In this way, the streaming data for detecting the status of each communication point, such as detecting an ensemble state, and the streaming data to be controlled by the communication method may not be the same.
[0089] The ensemble state may be detected by using both the comparison of image feature quantities according to this modified example and the comparison of sound feature quantities according to the embodiment. In the case of using both, the ensemble state may be detected when either one of the conditions is satisfied, or when both of the conditions are satisfied.
[0090] (10) When the electronic musical instrument 30 is played, either one of the sound generation control data Dc and the sound data Da output from the electronic musical instrument 30 may be transmitted from the communication terminal 20. For example, when the electronic musical instrument 30 is played at the communication point T1, if the communication points T2 and T3 have a function to convert the sound generation control data Dc into sound data Da (for example, if the electronic musical instrument 30 is provided), the communication terminal 20 at the communication point T1 does not transmit the sound data Da generated by the electronic musical instrument 30. On the other hand, if the communication points T2 and T3 do not have a function to convert the sound generation control data Dc into sound data Da, the communication terminal 20 at the communication point T1 does not transmit the sound generation control data Dc generated by the electronic musical instrument 30. Information on whether or not the communication points have a function to convert the sound generation control data Dc into sound data Da may be transmitted in advance from each communication point to the communication management server 1. In this manner, unnecessary streaming data is not transmitted from each communication point.
[0091] (11) Although the ensemble state has been detected based on the relationship between the sounding periods at a plurality of communication points, it may be detected based on the correlation between sounds instead of the sounding periods. In this case, it is only necessary to determine whether the sounds at a plurality of communication points have a predetermined correlation in each of the sections W1 to W5. The feature amount may be any information necessary to acquire the correlation, and may be any information related to the sounds. For example, the feature amount as the information related to the sounds may be BPM (Beats Per Minute) calculated based on the strength of the sounds. In this case, the ensemble state may be detected when the error in the BPM at each communication point is 10% or less in the sections W1 to W5.
[0092] In the case where the same performance is performed at multiple communication sites, the sound waveform signals in the sound data Da transmitted from the multiple communication sites may be compared to obtain the degree of agreement of the sound waveform signals as a correlation. Therefore, the feature amount as information related to the sound is the sound waveform signal. Taking into account communication delays, the degree of agreement of the sound waveform signals is calculated while changing the relative temporal positions of both signals within a predetermined range, and the value at which the degree of agreement is greatest is adopted as the degree of agreement. If this degree of agreement is greater than a predetermined value, an ensemble state may be detected.
[0093] As another example of a method for taking communication delay into consideration, the communication management server 1 may acquire a communication delay time when receiving sound data Da from each communication base, and adjust the time position of the sound data Da from the communication base based on the communication delay time to the timing of transmission from the communication base. In this way, by adjusting the time position of the sound data Da corresponding to each communication base, it is possible to convert the sound data Da based on the time received by the communication management server 1 into sound data Da based on the time transmitted from the communication base.
[0094] The sound data Da converted in this way may be used to compare the sound waveform signals to calculate the degree of match between the sound waveform signals. This makes it possible to reduce the effect of communication delay time. Depending on the accuracy of the acquired communication delay time, the above method may be used in combination to calculate the degree of match between the sound waveform signals while changing the time position within a predetermined range for both signals based on the converted sound data Da.
[0095] Other musical analysis elements such as pitch, volume, timbre, playing style, chord, etc. may be used as feature quantities as information related to sound. An ensemble state may be detected not only by comparing elements of the same type, but also by comparing elements of different types. For example, a chord and a pitch may be compared, and a pitch that is highly in harmony with the constituent notes of the chord may be set as a condition for detecting an ensemble state.
[0096] For example, for instruments that can simultaneously produce multiple notes such as chords (e.g., keyboard instruments such as the piano, and string instruments such as the guitar), chord is used as a feature. On the other hand, for instruments that produce single notes (e.g., string instruments such as the violin, and wind instruments such as the saxophone and trumpet), pitch is used as a feature. By comparing the chord and pitch, the ensemble state of both instruments may be detected. For human voices rather than instruments, it may be used as a comparison target in detecting the ensemble state. If it is one voice, it corresponds to an instrument that produces a single note, and if it is multiple voices, it corresponds to an instrument that can simultaneously produce multiple notes.
[0097] When pitch is used, for example, the ensemble state may be detected by using the change in pitch over time, i.e., the phrase. In this case, the ensemble state may be detected on the condition that the ratio of the same pitch in the two phrases to be compared is equal to or greater than a predetermined ratio.
[0098] (12) The communication method for transmitting each data is not limited to one selected from P2P communication and client-server communication, but may be a combination of communication methods with different communication delays. If desired, various combinations may be selected.
[0099] (13) The detection of the ensemble state is not limited to being performed by the communication management server 1, but may be performed by the communication terminal 20 at any of the communication bases. In this case, the communication terminal 20 detects the ensemble state based on a comparison of data transmitted from the other communication bases. If the communication terminal 20 detects the ensemble state as a result of the comparison, it only needs to transmit to the communication management server 1 a result that the ensemble state has been detected.
[0100] The above is a description of the modified example.
[0101] According to one embodiment of the present invention, there is provided a communication control method, which includes: acquiring a comparison result between a first feature amount in first streaming data transmitted from a first terminal by a first communication method and a second feature amount in second streaming data transmitted from a second terminal by the first communication method; and, if the first feature amount and the second feature amount have a first relationship based on the comparison result, transmitting a first switching signal to the first terminal and the second terminal for switching streaming data to be communication-controlled, transmitted between the first terminal and the second terminal by the first communication method, from transmission by the first communication method to transmission by a second communication method different from the first communication method. The method may further be configured as follows.
[0102] In the communication control method, the first feature and the second feature include information specifying a pronunciation period, the first relationship includes that a first pronunciation period defined by the first feature and a second pronunciation period defined by the second feature exist in each of a predetermined number of determination periods, and the streaming data to be communication controlled includes the first streaming data and the second streaming data.
[0103] In the communication control method, the first feature and the second feature include information related to sound, the first relationship includes that the first feature and the second feature have a predetermined correlation, and the streaming data to be communication controlled includes the first streaming data and the second streaming data.
[0104] In the communication control method, the information related to the sound is a sound waveform signal, and the first relationship includes that the degree of sound matching obtained while changing the relative temporal position between the sound waveform signal corresponding to the first feature and the sound waveform signal corresponding to the second feature within a predetermined range is greater than a predetermined value.
[0105] In the communication control method, the information related to the sound is a sound waveform signal, and the first relationship includes a degree of match obtained between a sound waveform signal corresponding to the first feature whose time position has been adjusted based on a communication delay time for the first terminal and a sound waveform signal corresponding to the second feature whose time position has been adjusted based on a communication delay time for the second terminal, the degree of match being greater than a predetermined value.
[0106] In the communication control method, a communication delay in the second communication method is smaller than a communication delay in the first communication method.
[0107] In the communication control method, in the first communication method, the first terminal and the second terminal communicate with each other via a server, and in the second communication method, the first terminal and the second terminal communicate with each other by P2P.
[0108] In the communication control method, third streaming data including sound waveform signal data is further transmitted from the first terminal to the second terminal by the first communication method, and the first streaming data is The streaming data includes sound generation control data, the streaming data to be communication controlled includes the first streaming data, and even when the first feature and the second feature have a first relationship, the third streaming data is transmitted by the first communication method.
[0109] In the communication control method, third streaming data including sound waveform signal data is further transmitted from the first terminal to the second terminal via the first communication method, the first streaming data includes sound generation control data, the streaming data to be controlled by the communication includes the first streaming data and the third streaming data, and when the first terminal or the communication load of the first terminal satisfies a predetermined condition, a second switching signal for switching the third streaming data from transmission via the second communication method to transmission via the first communication method is transmitted to the first terminal.
[0110] In the communication control method, third streaming data including sound waveform signal data is further transmitted from the first terminal to the second terminal via the first communication method, the first streaming data includes sound generation control data, the streaming data to be controlled by the communication includes the first streaming data and the third streaming data, and when a processing load of the first terminal or the second terminal satisfies a predetermined condition, a second switching signal for switching the third streaming data from transmission via the second communication method to transmission via the first communication method is transmitted to the first terminal.
[0111] In the communication control method, when the first feature and the second feature have a second relationship after the first feature and the second feature have a first relationship based on the comparison result, a second switching signal is transmitted to the first terminal and the second terminal to release the establishment of communication by the second communication method and switch the streaming data to be communication-controlled from transmission by the second communication method to transmission by the first communication method.
[0112] In the communication control method, when the first feature amount and the second feature amount have a first relationship, the streaming data of the communication control target is further transmitted to a third terminal by the second communication method. [Explanation of symbols]
[0113] 1...communication management server, 11...control unit, 13...storage unit, 15...communication unit, 20...communication terminal, 21...control unit, 23...storage unit, 25...communication unit, 27...operation unit, 30...electronic musical instrument, 40...sound collection device, 50...imaging device, 60...sound emission device, 70...display device, 80...SFU server
Claims
1. acquiring first streaming data from a first terminal at a first communication point by a first communication method and second streaming data from a second terminal at a second communication point by the first communication method; detecting whether or not an ensemble state in which performance is being performed simultaneously at the first communication location and the second communication location is being performed based on the first streaming data and the second streaming data; when the ensemble state is detected, acquisition of at least a portion of the first streaming data and at least a portion of the second streaming data is switched to a second communication method having a smaller communication delay than the first communication method. Communications control method.
2. the first streaming data and the second streaming data are data including at least a sound generation start timing, The communication control method according to claim 1 , wherein the ensemble state is detected based on the sound production start timing.
3. the first streaming data and the second streaming data further include a sound output end timing, The communication control method according to claim 2 , wherein the ensemble state is detected based on a sound generation period determined from the sound generation start timing and the sound generation end timing.
4. The communication control method according to claim 3 , wherein the sound generation period is a period from the sound generation start timing to the sound generation end timing.
5. 4. The communication control method according to claim 2, wherein the ensemble state is detected when the sound generation period occurs at the first communication location and the second communication location within a predetermined determination period.
6. 4. The communication control method according to claim 2, wherein the ensemble state is detected when the sound generation periods exist in one section even if the sound generation periods at a plurality of communication points do not overlap.
7. Detecting a non-ensemble state in which no performance is taking place at the first communication base station and the second communication base station, or a state in which performance is taking place at only one of the first communication base station and the second communication base station, The communication control method according to claim 1 , further comprising the step of switching from the second communication system to the first communication system when the non-ensemble state is detected.
8. the first streaming data includes sound control data or sound data and video data; The communication control method according to claim 1 , wherein, when the ensemble state is detected, the sound generation control data or the sound data is acquired by a second communication method, and the video data is acquired by a first communication method.
9. acquiring first streaming data from a first terminal at a first communication point by a second communication method and second streaming data from a second terminal at a second communication point by the second communication method; detect a non-ensemble state, which is a state in which no performance is taking place at the first communication location and the second communication location, or a state in which performance is taking place at only one of the first communication location and the second communication location, based on the first streaming data and the second streaming data; When the non-ensemble state is detected, the communication method is switched to a first communication method having a longer communication delay than the second communication method. Communications control method.
10. The communication control method according to claim 1 or 9, wherein the second communication method is P2P communication.
11. means for acquiring first streaming data from a first terminal at a first communication point by a first communication method, and second streaming data from a second terminal at a second communication point by the first communication method; a means for detecting whether or not an ensemble state in which performance is being performed simultaneously at the first communication location and the second communication location is being performed based on the first streaming data and the second streaming data; a means for switching, when the ensemble state is detected, for acquiring at least a portion of the first streaming data and at least a portion of the second streaming data to a second communication method having a smaller communication delay than the first communication method; A communication control system comprising:
12. means for acquiring first streaming data from a first terminal at a first communication point by a second communication method, and second streaming data from a second terminal at a second communication point by the second communication method; a means for detecting a non-ensemble state, which is a state in which no performance is taking place at the first communication location and the second communication location, or a state in which performance is taking place at only one of the first communication location and the second communication location, based on the first streaming data and the second streaming data; a means for switching to a first communication method having a longer communication delay than the second communication method when the non-ensemble state is detected; A communication control system comprising:
Citation Information
Patent Citations
Music session system utilizing communication network, and user terminal used for the music session system
JP2003323173A
Communication device, communication method, and program
JP2004294519A
Radio communication system and program for making computer perform transmitting and receiving of data in it
JP2005123791A
Radio communication device, radio communication system and radio communication method
JP2013110478A
Musical performance recording system
JP2014153515A
