Communication control method and communication control system

Abstract

To switch a communication system according to a situation between a plurality of communication bases.SOLUTION: A communication control method according to one embodiment includes: acquiring a result of comparison between a first feature quantity of first streaming data transmitted by a first communication system from a first terminal and a second feature quantity of second streaming data transmitted by the first communication system from a second terminal; and transmitting, when the first feature quantity and the second feature quantity has a first relation based upon the result of comparison, a first switching signal from the first terminal and second terminal, the first switching signal allowing streaming data as an object of communication control transmitted by the first communication system between the first terminal and the second terminal to be switched from the transmission by the first communication system to transmission by a second communication system different from the first communication system.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a technique for controlling communication. [Background technology]

[0002] Technologies have been developed that enable musical ensembles to be played even in distant locations by connecting multiple communication points where musical instruments are played via a network. However, communication delays caused by network connections make playing an ensemble difficult. Therefore, it is desirable to create an environment that minimizes communication delays, but communication delays cannot be eliminated. For this reason, a technology for reducing the effects of delays when communication delays exist is disclosed, for example, in Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-195982 Summary of the Invention [Problem to be solved by the invention]

[0004] Methods for communicating in real time between multiple communication points (multiple communication terminals) include, for example, peer-to-peer communication (hereinafter referred to as P2P communication) and client-server communication. P2P communication is a method in which communication terminals at each communication point connect on an equal basis. Client-server communication is a method in which communication terminals at each communication point connect via a server. Due to these differences in communication methods, P2P communication can reduce communication delays, but is prone to increase the processing load and communication volume of communication terminals. On the other hand, client-server communication can suppress increases in the load and communication volume of communication terminals, but increases communication delays. Such servers have functions such as an SFU (Selective Forwarding Unit) and an MCU (Multipoint Control Unit), for example.

[0005] In the above-mentioned ensemble, communication delays must be kept to a minimum. Therefore, it is desirable to use P2P communication as the communication method when performing an ensemble. However, if the processing load on the communication terminal becomes large, the processing load may result in large delays or may affect other processing on the communication terminal. On the other hand, depending on the situation at each communication point, it may not necessarily be necessary to use P2P communication.

[0006] One of the objects of the present invention is to switch communication methods depending on the situation between a plurality of communication points. [Means for solving the problem]

[0007] According to one embodiment of the present invention, there is provided a communication control method, which includes: obtaining a comparison result between a first feature in first streaming data transmitted from a first terminal using a first communication method and a second feature in second streaming data transmitted from a second terminal using the first communication method; and, if the first feature and the second feature have a first relationship based on the comparison result, transmitting a first switching signal to the first terminal and the second terminal to switch streaming data that is subject to communication control and that is transmitted between the first terminal and the second terminal using the first communication method from transmission using the first communication method to transmission using a second communication method that is different from the first communication method. [Effects of the Invention]

[0008] According to the present invention, the communication method can be switched depending on the situation between a plurality of communication points. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating the configuration of a communication system according to an embodiment of the present invention. [Figure 2] FIG. 3 is a diagram illustrating a communication control table according to an embodiment of the present invention. [Figure 3]10 is a flowchart illustrating a communication switching method of a communication terminal according to an embodiment of the present invention. [Figure 4] 10 is a flowchart illustrating a communication control method of a management server according to an embodiment of the present invention. [Figure 5] 10A and 10B are diagrams illustrating a method for detecting an ensemble state in one embodiment of the present invention. [Figure 6] 10A and 10B are diagrams illustrating a method for detecting an ensemble state in one embodiment of the present invention. [Figure 7] FIG. 10 is a diagram illustrating a method for controlling a communication mode in an embodiment of the present invention. [Figure 8] FIG. 2 is a diagram illustrating the flow of data between communication points. [Figure 9] FIG. 2 is a diagram illustrating a method for synchronizing each piece of data. [Figure 10] 10A and 10B are diagrams illustrating a method for synchronizing each piece of data when video data Dv is not synchronized. [Figure 11] FIG. 10 is a diagram illustrating a method for synchronizing each piece of data in a specific mode. DETAILED DESCRIPTION OF THE INVENTION

[0010] A communication system according to one embodiment of the present invention will be described in detail below with reference to the drawings. The embodiment described below is merely an example of an embodiment of the present invention, and the present invention should not be construed as being limited to these embodiments. In the drawings referred to in this embodiment, identical parts or parts having similar functions are designated by the same or similar reference symbols (reference symbols consisting of a number followed by A, B, etc.), and repeated explanations of such parts may be omitted.

[0011] [1. Communication system configuration] FIG. 1 is a diagram illustrating the configuration of a communication system according to one embodiment of the present invention. The communication system includes a communication management server 1 connected to a network NW such as the Internet. The communication management server 1 controls communication between multiple communication points connected to the network NW according to the status of each communication point. The functions of the communication management server 1, which will be described later, may be realized by multiple servers working together. In this example, two communication methods are used for communication between the communication points. The two communication methods are P2P communication and client-server communication. Although FIG. 1 illustrates three communication points T1, T2, and T3, the number is not limited to this. When the communication points T1, T2, and T3 are not distinguished from each other, they are simply referred to as communication points. The SFU server 80 is used in client-server communication and acts as an intermediary between the communication points.

[0012] A communication terminal 20 is located at each communication point. An electronic musical instrument 30, a sound collection device 40, an imaging device 50, a sound emission device 60, and a display device 70 are connected to the communication terminal 20. Although a communication terminal 20 is always present at each communication point, at least one of the electronic musical instrument 30, the sound collection device 40, the imaging device 50, the sound emission device 60, and the display device 70 does not have to be connected to the communication terminal 20. At least one of the electronic musical instrument 30, the sound collection device 40, the imaging device 50, the sound emission device 60, and the display device 70 may be configured as an integrated device with the communication terminal 20.

[0013] The electronic musical instrument 30 includes performance controls, a sound source that outputs sound data Da in response to operations on the performance controls, and a data output unit that outputs sound generation control data Dc corresponding to these operations. In this example, the electronic musical instrument 30 is an electronic piano having keys as performance controls. The sound data Da and sound generation control data Dc are output to the communication terminal 20. The sound data Da is data representing a sound waveform signal, and may be output to the sound output device 60. The sound generation control data Dc is, for example, M This is data for controlling a sound source in accordance with a predetermined standard such as the IDI standard. In this case, the sound generation control data Dc includes information that defines the sound generation content, such as information regarding the sound generation start timing (note on) and information regarding the sound generation end timing (note off).

[0014] The sound source can also output sound data Da under the control of the sound generation control data Dc. The electronic musical instrument 30 may include a driver such as a solenoid that drives the performance controls in response to the sound generation control data Dc input from an external device. When the sound generation control data Dc is received from another communication point, the sound generation control data Dc is supplied to such a sound source or driver. Driving the performance controls creates a situation equivalent to a performance being performed on the electronic musical instrument 30, and sound data Da is generated in response to the sound generation control data Dc.

[0015] The sound collection device 40 has, for example, a microphone or an input terminal for a sound waveform signal, and outputs the sound input to the microphone or the sound waveform signal input to the input terminal to the communication terminal 20 as sound data Da.

[0016] The imaging device 50 has, for example, a camera, and outputs video data Dv corresponding to images captured by the camera to the communication terminal 20. In the following description, the term "image" includes both still images and video images.

[0017] The sound emitting device 60 has, for example, a speaker, and outputs from the speaker a sound indicated by the sound data Da supplied from the communication terminal 20. The sound data Da supplied to the sound emitting device 60 is sound data Da transmitted from another communication point, but may also include sound data Da generated at the sound emitting device 60's own communication point.

[0018] The display device 70 has, for example, a liquid crystal display, and displays on the liquid crystal display an image indicated by the video data Dv supplied from the communication terminal 20. The video data Dv supplied to the display device 70 is video data Dv transmitted from another communication base, but may also include video data Dv generated at the display device's own communication base.

[0019] [2. Configuration of communication terminal] The communication terminal 20 includes a control unit 21, a storage unit 23, a communication unit 25, and an operation unit 27. The control unit 21 includes a CPU, RAM, ROM, etc. The control unit 21 performs processing in accordance with instructions defined in a program stored in the storage unit 23 by executing the program with the CPU. In this example, a program for performing processing to realize a method for switching communications (a communication switching method) is stored in the storage unit 23. The communication switching method includes processing for switching a communication method for transmitting data based on an instruction from the communication management server 1.

[0020] The communication unit 25 includes a communication module, connects to the network NW, and transmits and receives various data to and from external devices such as the communication management server 1, communication terminals 20 at other communication points, and the SFU server 80. Data transmitted between the communication unit 25 (communication terminal 20) and external devices other than the communication management server 1 includes sound generation control data Dc, sound data Da, and video data Dv, and is transmitted as streaming data. The communication method used to transmit this data is set under the control of the control unit 21. As described above, the communication method in this example includes P2P communication and client-server communication.

[0021] The operation unit 27 includes operation devices such as a mouse and a keyboard, accepts user operations on the operation devices, and outputs a signal corresponding to the operation to the control unit 21.

[0022] The storage unit 23 includes a storage device such as a nonvolatile memory, and is executed by the control unit 21. The storage unit 23 stores a program. In addition, various data used in the communication terminal 20 is also stored. The data stored in the storage unit 23 includes a communication control table. This program may be provided to the communication terminal 20 in a state where it is stored in a computer-readable recording medium such as a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, or a semiconductor memory, as long as it is executable by a computer. In this case, the communication terminal 20 may be provided with a device for reading the recording medium. The program may also be provided by downloading it via the communication unit 25.

[0023] 2 is a diagram illustrating a communication control table according to one embodiment of the present invention. The communication control table defines the communication method used to transmit sound generation control data Dc, sound data Da, and video data Dv in each of a plurality of communication modes. The plurality of communication modes include modes A, B, C, and D. The communication methods include client-server communication (SFU) and P2P communication (P2P).

[0024] In mode A, all data is transmitted via client-server communication (SFU). In mode D, all data is transmitted via P2P communication (P2P). On the other hand, in mode B, sound generation control data Dc is transmitted via P2P communication (P2P), and sound data Da and video data Dv are transmitted via client-server communication (SFU). In mode C, sound generation control data Dc and sound data Da are transmitted via P2P communication (P2P), and video data Dv is transmitted via client-server communication (SFU).

[0025] When communication terminal 20 is instructed by communication management server 1 to set a communication mode, control unit 21 refers to this communication control table to determine the communication method for transmitting each piece of data corresponding to that communication mode. In this way, control unit 21 sets the communication method corresponding to the instructed communication mode in communication unit 25. The setting of the communication mode by communication management server 1 will be described later.

[0026] [3. Communication switching method] Next, a description will be given of a communication switching method in communication terminal 20. As described above, each process in the communication switching method is executed by control unit 21. The process of the communication switching method described below is started by, for example, a user instruction input to operation unit 27.

[0027] 3 is a flowchart illustrating a communication switching method for a communication terminal according to one embodiment of the present invention. The communication terminal 20 connects to the communication management server 1 (step S110). At this time, the communication terminal 20 designates a session room for the communication base to which the connection is to be made to participate. A new session room can also be opened in the communication management server 1. The communication terminals 20 at the multiple communication bases participating in this session room are the terminals that will transmit data to each other.

[0028] The communication terminal 20 starts transmitting the feature information, communication load information, and processing load information to the communication management server 1 (step S120). While the communication terminal 20 is connected to the communication management server 1, it transmits the feature information, communication load information, and processing load information to the communication management server 1 at predetermined time intervals (for example, every 100 milliseconds).

[0029] The feature information includes the feature of the sound generation control data Dc and sound data Da transmitted as streaming data. In this example, the feature is information indicating whether sound is being generated at the time of transmission. Therefore, the feature information regarding the sound data Da indicates the sound generation period of the sound data Da by indicating that sound is being generated during the period when the volume level obtained from the sound data Da exceeds a predetermined value, and that sound is not being generated during other periods. The feature information regarding the sound generation control data Dc indicates the period from note-on indicated by the sound generation control data Dc to note-on indicated by the sound generation control data Dc. The feature amount information on the sound generation control data Dc may be configured to indicate that sound is being generated when the sound generation control data Dc is converted into sound data Da by a sound source and the volume level obtained from the sound data Da exceeds a predetermined value.

[0030] The communication load information is information that indicates the load of the communication line used when the communication terminal 20 transmits data. The communication load can be obtained, for example, by measuring the round trip time in P2P communication between the communication terminal 20 and another communication terminal 20. Generally, the amount of data transmitted and received is greater in P2P communication than in client-server communication, resulting in a greater communication load. Therefore, if there is no capacity in the communication line, low latency cannot be achieved even in P2P communication.

[0031] The processing load information is information indicating the processing load on the communication terminal 20. The processing load corresponds to, for example, the load on the CPU. Generally, the amount of data sent and received is greater in P2P communication than in client-server communication, and therefore the processing load increases due to encoding, decoding, etc. Therefore, if the communication terminal 20 does not have sufficient data processing capacity, low latency cannot be achieved even in P2P communication. This processing load may also include that caused by other programs running in parallel on the communication terminal 20.

[0032] When the communication terminal 20 connects to the communication management server 1, it receives an instruction to set a communication mode from the communication management server 1 and sets the communication mode (step S130). The communication mode specified by the communication management server 1 is the communication mode set for the session room specified by the communication terminal 20. If a new session room is opened, for example, and no communication mode has been set for this session room, the communication management server 1 instructs the communication terminal 20 to set communication mode A. If the session room has already been opened, another communication mode (mode B, mode C, or mode D) may have been set by a communication control method described later.

[0033] The communication terminal 20 refers to the communication control table and transmits each piece of data according to the set communication mode. Next, the communication terminal 20 waits until it receives a switching signal S(X) from the communication management server 1 (step S140; No). Here, X represents one of A, B, C, and D. A, B, C, and D correspond to mode A, mode B, mode C, and mode D, respectively. Upon receiving the switching signal S(X) (step S140; Yes), the communication terminal 20 switches the communication mode by setting the communication mode X according to this switching signal S(X) (step S150). Then, the communication terminal 20 waits until it receives the switching signal S(X) (step S140; No).

[0034] The communication terminal 20 is controlled by the communication management server 1 so as to change the communication method used for transmitting the sound generation control data Dc, the sound data Da, and the video data Dv.

[0035] [4. Communication Management Server Configuration] The communication management server 1 includes a control unit 11, a storage unit 13, and a communication unit 15. The control unit 11 includes a CPU, RAM, ROM, etc. The control unit 11 executes a program stored in the storage unit 13 using the CPU, thereby performing processing according to instructions defined in the program. This program includes a program for performing processing to realize a method for controlling communication of the communication terminal 20 (a communication control method). The communication control method includes processing for instructing the communication terminal 20 to switch the communication method for transmitting data. Details of this processing will be described later. The communication unit 15 includes a communication module, connects to the network NW, and transmits various types of data to and from the communication terminals 20 at each communication point.

[0036] The memory unit 13 includes a storage device such as a hard disk, and stores a program executed by the control unit 11. It also stores various other data used in the communication management server 1. This program need only be executable by a computer, and may be provided to the communication management server 1 in a state where it is stored on a computer-readable recording medium such as a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, or a semiconductor memory. In this case, the communication management server 1 only needs to be equipped with a device for reading the recording medium. This program may also be provided by downloading it via the communication unit 15.

[0037] [5. Communication control method] Next, a communication control method in the communication management server 1 will be described. As described above, each process in the communication control method is executed by the control unit 11. The process of the communication control method described below is started, for example, when multiple communication points join a session room. At this time, the performance mode is set to mode A. Therefore, the transmission of sound generation control data Dc, sound data Da, and video data Dv between each communication point is all set as client-server communication.

[0038] 4 is a flowchart illustrating a communication control method of the management server according to an embodiment of the present invention. The communication management server 1 compares feature information received from the communication terminals 20 at a plurality of communication points and waits until an ensemble state is detected (step S210; No).

[0039] An ensemble state refers to a state in which a performance is taking place at multiple communication points. The performance at each communication point may be, for example, a performance using an electronic musical instrument 30 or a performance using an acoustic musical instrument. When a performance is using an electronic musical instrument 30, at least one of sound generation control data Dc and sound data Da is supplied to the communication terminal 20. When a performance is using an acoustic musical instrument, the sound generated by the acoustic musical instrument is input to a sound collection device 40, and the sound data Da is supplied to the communication terminal 20.

[0040] As described above, the feature information transmitted from the communication terminals 20 at each communication point includes information indicating whether the sound generation control data Dc and sound data Da transmitted as streaming data are being generated at each time. Therefore, by comparing this feature information along the time axis, it is possible to detect whether sounds are being generated simultaneously at multiple communication points. However, even if the feature information can detect the presence of a sound, it is possible that the sound is not necessarily generated as a result of musical performance. The method for detecting the ensemble state in this example is a method for excluding sounds not generated as a result of musical performance as much as possible.

[0041] 5 and 6 are diagrams illustrating a method for detecting an ensemble state in one embodiment of the present invention. These diagrams show the sound generation period (mesh portion in the diagram) at each communication point along time t, based on feature information transmitted from the communication terminals 20 at each of the communication points T1, T2, and T3. The feature information indicates the generation of sound generation control data Dc at communication point T1, and the feature information indicates the generation of sound data Da at communication points T2 and T3. FIG. 6 corresponds to the situation after a short time has passed since the situation in FIG. 5.

[0042] A specific example of an ensemble detection method will be described. Five intervals W1 to W5 are defined by dividing the period from five seconds before the ensemble detection determination time DP (current time) to the determination time DP into one-second intervals. The communication management server 1 determines whether there is a sound generation period at multiple communication points in each interval (determination period).

[0043] In an actual ensemble, the sounding periods do not overlap, that is, there are cases where there are no simultaneous sounds. Therefore, even if the sounding periods at multiple communication points do not overlap, the sounding periods in one section If there is an ensemble sound in sections W1 to W5, the section is determined to be a section in which an ensemble sound was present. The "multiple communication points" referred to here may be different from all communication points participating in the session room, i.e., it refers to at least two of the communication points participating in the session room. When the communication management server 1 determines that an ensemble sound was present in sections W1 to W5, it detects that the session room in which these communication points participate has entered an ensemble state.

[0044] The time going back from the determination time DP (the total time of the sections W1 to W5), the number of divided sections, and the duration of each section are merely examples and may be set to various values. The lengths of the sections W1 to W5 may be different. In this case, the further the section is from the determination time DP, the shorter the section may be, or the closer the section is to the determination time DP, the shorter the section may be. The sections W1 to W5 do not have to be consecutive. In other words, adjacent sections may be spaced apart in time.

[0045] In Figure 5, since only communication point T1 produces sound in section W1, it is determined that there is no ensemble sound in section W1. On the other hand, it is determined that there is ensemble sound in sections W2 to W5. As described above, it is also determined that there is an ensemble sound in section W2, where the sound production period of communication point T1 and the sound production period of communication point T2 do not overlap. At this point, since it has been determined that there is no ensemble sound in section W1, it has not been detected that the session room in which communication points T1 to T3 participate has entered an ensemble state.

[0046] 6, it is determined that an ensemble sound is present in sections W1 to W5. Therefore, the communication management server 1 detects that the session room in which communication points T1 to T3 participate has entered an ensemble state. Once an ensemble state is detected, the session room will be maintained in the ensemble state until a non-ensemble state, which will be described later, is detected, even if the above conditions are no longer met.

[0047] Returning to Figure 4, the explanation will continue. When an ensemble state is detected (step S210; Yes), the communication management server 1 transmits a switching signal S(D) to the communication terminal 20 at each communication point (step S230). As described above, when the communication terminal 20 receives the switching signal S(D), the communication mode is switched from mode A to mode D. That is, the transmission of all sound generation control data Dc, sound data Da, and video data Dv between each communication point is switched from client-server communication to P2P communication.

[0048] The communication management server 1 compares the feature information received from the communication terminals 20 at the plurality of communication points and performs the communication mode switching process until a non-ensemble state is detected (step S310; No, step S400). A non-ensemble state refers to a state where no performance is taking place at any communication point or a state where performance is taking place at only one communication point.

[0049] A specific example of a method for detecting a non-ensemble state is the opposite of the method for detecting an ensemble state: a non-ensemble state is detected when it is determined that there are no ensemble sounds in any of the sections W1 to W5 corresponding to the determination time point DP.

[0050] When a non-ensemble state is detected (step S310; Yes), the communication management server 1 transmits a switching signal S(A) to the communication terminal 20 at each communication point (step S900). As described above, when the communication terminal 20 receives the switching signal S(A), the communication mode is set to mode A. That is, the transmission of the sound generation control data Dc, the sound data Da, and the video data Dv between each communication point is all set to client-server communication. The communication management server 1 then waits until an ensemble state is detected again (step S210; No).

[0051] The communication mode switching process (step S400) is a process of switching the communication mode in accordance with the load value Ld. The load value Ld is determined by the communication load information and the processing load transmitted from each communication terminal 20. It is an index of the magnitude of the load calculated by the communication management server 1 based on the load information. In this example, the communication load and processing load corresponding to the multiple communication terminals 20 each range from 0% to 100%, and the largest value is the load value Ld.

[0052] Fig. 7 is a diagram illustrating a method for controlling a communication mode in one embodiment of the present invention, showing conditions under which the communication mode is switched depending on the load value Ld in the communication mode switching process.

[0053] For example, when the communication mode is mode D and the load value Ld becomes larger than the threshold value TC1, the communication management server 1 transmits a switching signal S(C) to each communication terminal 20 to switch the communication mode to mode C. When the communication mode is mode C and the load value Ld becomes larger than the threshold value TB1, the communication management server 1 transmits a switching signal S(B) to each communication terminal 20 to switch the communication mode to mode B. On the other hand, when the communication mode is mode C and the load value Ld becomes smaller than the threshold value TC21, the communication management server 1 transmits a switching signal S(D) to each communication terminal 20 to switch the communication mode to mode D.

[0054] Here, TC1>TC2. By doing so, even if Ld becomes larger than TC1 and the communication mode switches to mode C, and then immediately after Ld becomes smaller than TC1, the mode does not immediately switch to mode D. The relationships of the other thresholds are similar: TB1>TB2 and TA1>TA2. The load of mode A is less than the load of mode B, which is less than the load of mode C, which is less than the load of mode D. Therefore, TA1, TB1, and TC1 may be set as the same threshold. TA2=TB2=TC2 may be set as the same threshold.

[0055] In this way, in the communication mode switching process (step S400), when the load value Ld increases, the communication management server 1 changes the communication mode to a mode that reduces the load compared to the current mode, and when the load value Ld decreases, the communication mode changes to a mode that reduces the communication delay compared to the current mode. This communication mode switching process continues until the non-ensemble state described above is detected.

[0056] [6. Data transmission and reception example] Next, an example of a communication method applied to the transmission of sound generation control data Dc, sound data Da, and video data Dv between communication points using the above-described communication control method and communication switching method will be described.

[0057] FIG. 8 is a diagram illustrating the flow of data between communication points. When no performance is taking place at each point, the communication mode remains set to Mode A. In this case, all of the sound generation control data Dc, sound data Da, and video data Dv are transmitted and received between the communication points via the SFU server 80 (paths indicated by dashed lines in FIG. 8). In other words, client-server communication is applied to all data. In this situation, even if communication delays are large, no particular problems arise because no ensemble is taking place, so reducing the communication load and processing load takes priority.

[0058] Assume that a performance is taking place at communication point T1. In this state, the communication mode remains in mode A. If a performance is then performed in communication mode T2, an ensemble state is detected. This causes the communication mode to switch to mode D. Therefore, all of the sound generation control data Dc, sound data Da, and video data Dv are transmitted and received directly between communication points via P2P communication (paths shown by solid lines in Figure 8).

[0059] After that, if the load value Ld increases, the communication mode is switched to mode C. Therefore, the sound generation control data Dc and sound data Da continue to be transmitted via P2P communication, but only the video data Dv is switched to transmission via the SFU server 80. When the load value Ld changes, the communication mode is switched in accordance with the change, and the communication method for each data is switched.

[0060] As data for minimizing communication delays, video data Dv has the lowest priority, and sound generation control data Dc has the highest priority. Even if the video data Dv is delayed, as long as the sound delay is small, it will not significantly hinder ensemble performance. Therefore, when the communication mode is switched from mode D to mode C, the transmission of video data Dv is switched to client-server communication. Since the amount of data for sound generation control data Dc is small, maintaining P2P communication is unlikely to increase the load. Therefore, in communication modes other than mode A, P2P communication is used for the transmission of sound generation control data Dc.

[0061] When the performance is interrupted at at least one of the communication points T1 and T2, a non-ensemble state is detected. This causes the communication mode to switch to mode A, and all of the sound control data Dc, sound data Da, and video data Dv are transmitted directly between the communication points via P2P communication.

[0062] [7. Synchronization between each data] Next, a method for synchronizing the sound output from the sound emitting device 60 with the video displayed on the display device 70 at a communication point that has received the sound generation control data Dc, sound data Da, and video data Dv will be described. Synchronization here refers to maintaining the temporal relationship between each part of the data at the data transmitting side at the data receiving side as well. The following description assumes that the data transmitting side is communication point T1 and the data receiving side is communication point T2.

[0063] Fig. 9 is a diagram illustrating a method for synchronizing each piece of data. In Fig. 9, time tp indicates the timing at which a specific portion of each piece of data is transmitted from communication point T1, and time te indicates the timing at which data received at communication point T2 is synchronized and output.

[0064] The time dc is a time equivalent to the communication delay of the sound generation control data Dc. Therefore, the communication point T2 receives the sound generation control data Dc after the time dc has elapsed since the time tp. The time da is a time equivalent to the communication delay of the sound data Da. Therefore, the communication point T2 receives the sound data Da after the time da has elapsed since the time tp. The time dv is a time equivalent to the communication delay of the video data Dv. Therefore, the communication point T2 receives the video data Dv after the time dv has elapsed since the time tp.

[0065] In this example, in order to synchronize with the video data Dv, which has the greatest communication delay, the time te corresponds to the reception time of the video data Dv. Therefore, at the communication point T2, the communication terminal 20 supplies the sound production control data Dc to the electronic musical instrument 30 after time htc (= time dv - time dc) has elapsed since receiving the sound production control data Dc. The sound production control data Dc is buffered in the communication terminal 20 until time htc has elapsed. The electronic musical instrument 30 converts the sound production control data Dc into sound data Da using a sound source and supplies the sound production data Da to the sound production device 60. As a result, a sound corresponding to the sound production control data Dc is output from the sound production device 60.

[0066] At the communication point T2, the communication terminal 20 receives the sound data Da, and then after time hta (=time dv-time da) has elapsed, supplies the sound data Da to the sound emitting device 60. The sound data Da is buffered in the communication terminal 20 until time hta has elapsed. As a result, sound corresponding to the sound data Da is output from the sound emitting device 60. At the communication point T2, when the communication terminal 20 receives the video data Dv, it supplies the video data Dv to the display device 70. As a result, an image corresponding to the video data Dv is displayed on the display device 70. In this way, Then, the sound corresponding to the sound generation control data Dc, the sound corresponding to the sound data Da, and the image corresponding to the video data Dv are synchronized.

[0067] The time htc and the time hta may be obtained in advance using known technology in relation to each communication point (in the example of Figure 9, the relationship from communication point T1 to communication point T2), or may be obtained from the time difference between timestamps obtained sequentially by assigning a timestamp corresponding to the transmission time to each piece of data.

[0068] The times htc and hta may also be obtained by other methods. For example, at communication point T1, a specific action by a performer is captured on video and a specific sound is recorded simultaneously with the specific action. The resulting sound data Da and video data Dv may be received at communication point T2, and the time hta may be obtained based on the time difference between the specific action and the specific sound. At communication point T1, sound production control data Dc and sound data Da are obtained by playing the electronic musical instrument 30. The sound production control data Dc and sound data Da may be received at communication point T2, and the time htc may be obtained based on the time difference between the note-on included in the sound production control data Dc and the sound production start timing of the sound data Da (corresponding to time htx shown in FIG. 10 ) and the time hta.

[0069] When the video data Dv is significantly delayed, particularly when the communication mode is mode C, the video data Dv may be excluded from synchronization.

[0070] FIG. 10 is a diagram illustrating a method for synchronizing each piece of data when the video data Dv is not synchronized. Unlike the example shown in FIG. 9, the communication point T2 receives the video data Dv transmitted at time tp after time te. In this example, synchronization is performed with the sound data Da, which has the largest communication delay among the data to be synchronized, so time te corresponds to the reception time of the sound data Da. Therefore, at the communication point T2, the communication terminal 20 supplies the sound production control data Dc to the electronic musical instrument 30 after time htx (= time da - time dc) has elapsed since receiving the sound production control data Dc. The electronic musical instrument 30 converts the sound production control data Dc into sound data Da using a sound source and supplies the sound production data Da to the sound production device 60. As a result, a sound corresponding to the sound production control data Dc is output from the sound production device 60.

[0071] At the communication point T2, when the communication terminal 20 receives the sound data Da, it supplies the sound data Da to the sound emitting device 60. As a result, a sound corresponding to the sound data Da is output from the sound emitting device 60. In this way, the sound corresponding to the sound control data Dc and the sound corresponding to the sound data Da are synchronized. On the other hand, the video data Dv is not synchronized with these data.

[0072] Next, assume that the electronic musical instrument 30 at communication point T2 has a driver that drives the performance controls based on the sound generation control data Dc. In this case, a drive time is required from when the sound generation control data Dc is supplied to the electronic musical instrument 30 until the performance controls are driven. Each piece of data may be synchronized taking this drive time into consideration. In such synchronization, a specific mode is set.

[0073] Fig. 11 is a diagram illustrating a method for synchronizing each piece of data in a specific mode. The time dm shown in Fig. 11 corresponds to the drive time required from when the sound generation control data Dc is supplied until the performance operators are driven. The time dm may be preset as a characteristic of the drive unit of the electronic musical instrument 30, or may be measured in advance.

[0074] In this example, the time it takes for the sound data Da to be output from the electronic musical instrument 30 in response to the sound generation control data Dc is the longest. Therefore, the time te corresponds to the timing at which the sound data Da is output from the electronic musical instrument 30, whose performance controls are driven in response to the sound generation control data Dc. Therefore, at the communication point T2, the communication terminal 20 supplies the sound data Da to the sound emitting device 60 after the time hty (= time dc + time dm - time da) has elapsed since receiving the sound data Da. As a result, a sound corresponding to the sound data Da is output from the sound emitting device 60.

[0075] At communication point T2, the communication terminal 20 supplies the video data Dv to the display device 70 after time htv (=time dc+time dm-time dv) has elapsed since receiving the video data Dv. As a result, an image corresponding to the video data Dv is displayed on the display device 70. In this way, the sound corresponding to the sound control data Dc, the sound corresponding to the sound data Da, and the image corresponding to the video data Dv are synchronized.

[0076] When the time until sound data Da is output from the electronic musical instrument 30 in response to the sound generation control data Dc is the longest, i.e., when time dm is long, the sound data Da and video data Dv do not need to be transmitted by P2P communication, depending on time da and time dv. For example, even if sound data Da and video data Dv are transmitted from communication point T1 by client-server communication, if they are received at communication point T2 before time te, the sound data Da and video data Dv do not need to be transmitted by P2P communication. Therefore, in this case, the communication mode may be set to mode B even if it is mode C or D.

[0077] As described above, in one embodiment, the communication system can switch communication methods depending on the situation between multiple communication points. Therefore, it is possible to control communication by applying a communication method that reduces communication delays when the performances at multiple communication points are in an ensemble state, and applying a communication method that reduces the communication load and processing load when the performances are not in an ensemble state. If the communication load or processing load becomes large, it is also possible to control the communication method for each type of data.

[0078] <Modification> Although one embodiment of the present invention has been described above, this embodiment can be modified in various ways as follows. Furthermore, the above-described embodiment and the modifications described below can also be applied in combination with each other.

[0079] (1) The communication management server 1 may have the functions of the SFU server 80. In this case, the communication management server 1 may detect the ensemble state using data transmitted by client-server communication instead of feature information.

[0080] (2) The switching signal transmitted from the communication management server 1 to the communication terminal 20 is a signal for switching the performance mode, but instead of specifying the performance mode after switching, it may be a signal for specifying the communication method for each data. In this case, it may be in a form that specifies the data for which the communication method needs to be changed. In this case, the communication terminal 20 does not need to have a communication control table.

[0081] (3) The timing at which P2P communication between communication points is established may be when P2P communication becomes necessary depending on the communication mode, or may be before the start of detection of an ensemble state. When there is no data to be transmitted via P2P communication, the establishment of P2P communication may be terminated.

[0082] (4) When the communication method is switched, the communication terminal 20 may output data that has been received in the communication method before the switch and that has been buffered in the communication terminal 20 to the electronic musical instrument 30, the sound output device 60, and the display device 70 until the data is received in the communication method after the switch.

[0083] (5) In the embodiment, the communication points T1 and T2 are performing together, so that the session For communication point T3, which was not included in the comparison when the room was detected as being in an ensemble state, a process different from the set communication mode may be executed. For example, since there is no need to minimize communication delays for data transmitted from communication point T3, client-server communication may be used.

[0084] (6) The communication mode may be limited to modes A and D. In this case, there is no need to switch the communication mode depending on the communication load and processing load.

[0085] (7) The video data Dv may be transmitted by client-server communication in all communication modes, and may be excluded from the streaming data whose communication method is to be controlled.

[0086] (8) When the communication load and processing load exceed a predetermined value, communication terminal 20 may execute processing to reduce these loads. For example, communication terminal 20 may reduce the load by processing to reduce the video size, the video resolution, the video frame frequency, the audio sampling frequency, the compression rate, and the audio channel reduction (converting stereo to mono). The adjustment ranges of these processing are predetermined. When processing to reduce the load is executed in this manner, the load value Ld described above will not increase until adjustment within the adjustment range is no longer possible.

[0087] (9) The ensemble state may be detected by comparing the video data Dv. In this case, the feature information includes a feature related to the video data Dv. This feature may be, for example, information indicating whether an instrument is included in the image or whether a moving person is included in the image, as a result of analyzing an image represented by the video data Dv. The ensemble state may be detected by using the period during which an instrument is included in the image or the period during which a moving person is included in the image, instead of the sounding period used in the above embodiments. As such, if the communication method between communication points is controlled according to the status of each communication point, various methods can be applied to detect the status of each communication point.

[0088] Even in this case, the video data Dv may be excluded from the streaming data to be controlled by the communication method, as in the above-mentioned modified example (7). In this way, the streaming data for detecting the status of each communication point, such as detecting the ensemble state, and the streaming data to be controlled by the communication method do not need to be the same.

[0089] The ensemble state may be detected by a combination of comparing image feature quantities according to this modified example and comparing sound feature quantities according to the embodiment. When both methods are used, the ensemble state may be detected when either one of the conditions is met, or when both conditions are met.

[0090] (10) When a performance is performed by the electronic musical instrument 30, either the sound generation control data Dc or the sound data Da output from the electronic musical instrument 30 may be transmitted from the communication terminal 20. For example, when a performance is performed by the electronic musical instrument 30 at communication point T1, if communication points T2 and T3 have the function of converting the sound generation control data Dc into sound data Da (e.g., if they are equipped with the electronic musical instrument 30), the communication terminal 20 at communication point T1 will not transmit the sound data Da generated by the electronic musical instrument 30. On the other hand, if communication points T2 and T3 do not have the function of converting the sound generation control data Dc into sound data Da, the communication terminal 20 at communication point T1 will not transmit the sound generation control data Dc generated by the electronic musical instrument 30. Information on whether or not the function of converting the sound generation control data Dc into sound data Da is capable of being transmitted from each communication point to the communication management server 1 in advance. This prevents unnecessary streaming data from being transmitted from each communication point.

[0091] (11) While the ensemble state has been detected based on the relationship between sound production periods at multiple communication points, it may also be detected based on the correlation between sounds rather than sound production periods. In this case, it is sufficient to determine whether the sounds at multiple communication points have a predetermined correlation in each of sections W1 to W5. The feature amount may be any information necessary to obtain the correlation, and may be any information related to the sound. For example, the feature amount as information related to the sound may be BPM (Beats Per Minute), calculated based on the strength of the sound. In this case, the ensemble state may be detected when the BPM at each communication point has an error of 10% or less in sections W1 to W5.

[0092] In a situation where the same performance is being performed at multiple communication locations, the sound waveform signals in the sound data Da transmitted from the multiple communication locations may be compared, and the degree of match of the sound waveform signals may be obtained as a correlation. Therefore, the feature quantity representing sound-related information is the sound waveform signal. Taking communication delays into account, the degree of match of the sound waveform signals is calculated while varying the relative temporal positions of the two signals within a predetermined range, and the value with the greatest degree of match is adopted as the degree of match. An ensemble state may be detected when this degree of match is greater than a predetermined value.

[0093] As another example of a method for taking communication delays into consideration, the communication management server 1 may acquire the communication delay times when receiving sound data Da from each communication point, and adjust the time position of the sound data Da from the communication point based on these communication delay times to the timing of transmission from the communication point. By adjusting the time position of the sound data Da corresponding to each communication point in this way, sound data Da based on the time received by the communication management server 1 can be converted to sound data Da based on the time transmitted from the communication point.

[0094] The sound data Da converted in this way may be used to compare the sound waveform signals and calculate the degree of match between the sound waveform signals. This makes it possible to reduce the influence of communication delay time. Depending on the accuracy of the acquired communication delay time, the above method may be used in combination to calculate the degree of match between the sound waveform signals of both signals based on the converted sound data Da while changing the time position within a predetermined range.

[0095] Other musical analysis elements such as pitch, volume, timbre, playing style, and chords may also be used as feature quantities for sound-related information. Ensemble states are not limited to being detected by comparing elements of the same type, but may also be detected by comparing elements of different types. For example, chords and pitches may be compared, and a pitch that is highly harmonized with the constituent notes of the chord may be used as a condition for detecting an ensemble state.

[0096] For example, for instruments that can simultaneously produce multiple notes, such as chords (e.g., keyboard instruments such as the piano, and string instruments such as the guitar), chord is used as a feature. On the other hand, for instruments that produce single notes (e.g., string instruments such as the violin, and wind instruments such as the saxophone and trumpet), pitch is used as a feature. By comparing chord and pitch, the ensemble state of both instruments may be detected. For human voices rather than instruments, chord and pitch may be used as a comparison target in detecting the ensemble state. A single voice corresponds to an instrument that produces a single note, while multiple voices correspond to an instrument that can simultaneously produce multiple notes.

[0097] When pitch is used, for example, an ensemble state may be detected by using a change in pitch over time, i.e., a phrase, and the ensemble state may be detected on the condition that the proportion of identical pitches in the two phrases being compared is equal to or greater than a predetermined proportion.

[0098] (12) The communication method for transmitting each data is not limited to a P2P communication method or a client-server communication method, but may be a combination of communication methods with different communication delays. If desired, various combinations may be selected.

[0099] (13) The detection of the ensemble state may not only be performed by the communication management server 1, but also by the communication terminal 20 at one of the communication centers. In this case, the communication terminal 20 detects the ensemble state based on a comparison of data transmitted from other communication centers. If the communication terminal 20 detects the ensemble state as a result of the comparison, it may transmit to the communication management server 1 a result that the ensemble state has been detected.

[0100] The above is the explanation regarding the modified example.

[0101] According to one embodiment of the present invention, there is provided a communication control method comprising: obtaining a comparison result between a first feature amount in first streaming data transmitted from a first terminal using a first communication method and a second feature amount in second streaming data transmitted from a second terminal using the first communication method; and, if a first relationship is found between the first feature amount and the second feature amount based on the comparison result, transmitting a first switching signal to the first terminal and the second terminal for switching streaming data to be communication-controlled and transmitted between the first terminal and the second terminal using the first communication method from transmission using the first communication method to transmission using a second communication method different from the first communication method.

[0102] In the communication control method, the first feature and the second feature include information specifying a sounding period, the first relationship includes that a first sounding period specified by the first feature and a second sounding period specified by the second feature exist in each of a predetermined number of determination periods, and the streaming data to be communication controlled includes the first streaming data and the second streaming data.

[0103] In the communication control method, the first feature and the second feature include information related to sound, the first relationship includes that the first feature and the second feature have a predetermined correlation, and the streaming data to be communication controlled includes the first streaming data and the second streaming data.

[0104] In the communication control method, the information related to the sound is a sound waveform signal, and the first relationship includes that the degree of sound matching obtained while changing the relative temporal position between the sound waveform signal corresponding to the first feature and the sound waveform signal corresponding to the second feature within a predetermined range is greater than a predetermined value.

[0105] In the communication control method, the information related to the sound is a sound waveform signal, and the first relationship includes that the degree of match obtained between the sound waveform signal corresponding to the first feature whose time position has been adjusted based on the communication delay time for the first terminal and the sound waveform signal corresponding to the second feature whose time position has been adjusted based on the communication delay time for the second terminal is greater than a predetermined value.

[0106] In the communication control method, a communication delay in the second communication method is smaller than a communication delay in the first communication method.

[0107] In the communication control method, in the first communication method, the first terminal and the second terminal communicate with each other via a server, and in the second communication method, the first terminal and the second terminal communicate with each other by P2P.

[0108] In the communication control method, third streaming data including sound waveform signal data is further transmitted from the first terminal to the second terminal by the first communication method, and the first streaming data is The streaming data includes sound control data, the streaming data to be controlled in communication includes the first streaming data, and even when the first feature and the second feature have a first relationship, the third streaming data is transmitted by the first communication method.

[0109] In the communication control method, third streaming data including sound waveform signal data is further transmitted from the first terminal to the second terminal using the first communication method, the first streaming data includes sound generation control data, the streaming data to be communication controlled includes the first streaming data and the third streaming data, and when the communication load of the first terminal or the first terminal satisfies a predetermined condition, a second switching signal is transmitted to the first terminal to switch the third streaming data from transmission using the second communication method to transmission using the first communication method.

[0110] In the communication control method, third streaming data including sound waveform signal data is further transmitted from the first terminal to the second terminal using the first communication method, the first streaming data includes sound generation control data, the streaming data to be communication controlled includes the first streaming data and the third streaming data, and when the processing load of the first terminal or the second terminal satisfies a predetermined condition, a second switching signal is transmitted to the first terminal to switch the third streaming data from transmission using the second communication method to transmission using the first communication method.

[0111] In the communication control method, if the first feature and the second feature have a second relationship after the first feature and the second feature have a first relationship based on the comparison result, the establishment of communication by the second communication method is terminated and a second switching signal is transmitted to the first terminal and the second terminal to switch the streaming data to be communication-controlled from transmission by the second communication method to transmission by the first communication method.

[0112] In the communication control method, when the first feature amount and the second feature amount have a first relationship, the streaming data to be communication-controlled is further transmitted to a third terminal in the second communication method. [Explanation of symbols]

[0113] 1...communication management server, 11...control unit, 13...storage unit, 15...communication unit, 20...communication terminal, 21...control unit, 23...storage unit, 25...communication unit, 27...operation unit, 30...electronic musical instrument, 40...sound collection device, 50...imaging device, 60...sound emission device, 70...display device, 80...SFU server

Claims

1. acquiring first streaming data from a first terminal at a first communication point by a first communication method and second streaming data from a second terminal at a second communication point by the first communication method; detecting, based on the first streaming data and the second streaming data, whether or not an ensemble state in which performance is being performed simultaneously at the first communication point and the second communication point is being established; when the ensemble state is detected, acquisition of at least a portion of the first streaming data and at least a portion of the second streaming data is switched to a second communication method having a smaller communication delay than the first communication method. Communication control method.

2. the first streaming data and the second streaming data are data including at least a sound generation start timing; The communication control method according to claim 1 , wherein the ensemble state is detected based on the timing at which sound is produced.

3. the first streaming data and the second streaming data further include sound generation end timing; 3. The communication control method according to claim 2, wherein the ensemble state is detected based on a sound generation period determined from the sound generation start timing and the sound generation end timing.

4. The communication control method according to claim 3 , wherein the sound generation period is a period from the sound generation start timing to the sound generation end timing.

5. 4. The communication control method according to claim 3, wherein the ensemble state is detected when the tone generation period occurs at the first communication location and the second communication location within a predetermined determination period.

6. 4. The communication control method according to claim 3, wherein the ensemble state is detected when the sound generation periods at a plurality of communication points exist in one section even if the sound generation periods at the plurality of communication points do not overlap.

7. Detecting a non-ensemble state, which is a state in which no performance is taking place at the first communication base station and the second communication base station, or a state in which performance is taking place at only one of the first communication base station and the second communication base station; The communication control method according to claim 1 , wherein when the non-ensemble state is detected, the communication method is switched from the second communication method to the first communication method.

8. the first streaming data includes sound control data or sound data and video data; The communication control method according to claim 1 , wherein, when the ensemble state is detected, the sound generation control data or the sound data is acquired by a second communication method, and the video data is acquired by a first communication method.

9. acquiring first streaming data from a first terminal at a first communication point by a second communication method and second streaming data from a second terminal at a second communication point by the second communication method; a state in which no performance is being performed at the first communication point and the second communication point based on the first streaming data and the second streaming data, or A non-ensemble state is detected in which performance is being performed only at either the communication point or the second communication point; When the non-ensemble state is detected, the communication system is switched to a first communication system having a longer communication delay than the second communication system. Communication control method.

10. The communication control method according to claim 1 or claim 9, wherein the second communication method is P2P communication.

11. means for acquiring first streaming data from a first terminal at a first communication point by a first communication method and second streaming data from a second terminal at a second communication point by the first communication method; means for detecting, based on the first streaming data and the second streaming data, whether or not an ensemble state in which performance is being performed simultaneously at the first communication point and the second communication point is being established; a means for switching, when the ensemble state is detected, acquisition of at least a portion of the first streaming data and at least a portion of the second streaming data to a second communication method having a smaller communication delay than the first communication method; A communication control system, including:

12. means for acquiring first streaming data from a first terminal at a first communication point by a second communication method and second streaming data from a second terminal at a second communication point by the second communication method; means for detecting a non-ensemble state, which is a state in which no performance is taking place at the first communication point and the second communication point, or a state in which performance is taking place at only one of the first communication point and the second communication point, based on the first streaming data and the second streaming data; a means for switching to a first communication method having a longer communication delay than the second communication method when the non-ensemble state is detected; A communication control system, including:

Images