Media processing device, media processing method and media processing program

Abstract

In one embodiment of the present invention, a media processing device is a device at a first location and comprises: a reception unit that receives a packet in which is stored a second media acquired at a second location at a time at which a first media is generated at the second location, the first media having been acquired at a first time at the first location; and a processing unit that generates a third media from the second media in accordance with a processing mode based on the first time and a second time associated with reception of the packet in which the second media is stored, and outputs the third media to a presentation device.

Description

[Technical field]

[0001] One aspect of the present invention relates to a media processing device, a media processing method, and a media processing program. [Background technology]

[0002] In recent years, video and audio playback devices have come into use, which digitize video and audio filmed and recorded at a certain location, transmit them to a remote location in real time via communication lines such as an IP (Internet Protocol) network, and play the video and audio at the remote location. For example, public viewing, in which video and audio of a sports competition held at a sports venue or video and audio of a music concert held at a concert venue are transmitted to a remote location in real time, is being actively carried out. Such video and audio transmission is not limited to one-to-one one-way transmission. Bidirectional transmission is also performed, in which video and audio are transmitted from a venue where a sports competition is held (hereinafter referred to as an event venue) to multiple remote locations, and the video and audio of spectators enjoying the event and the audio such as cheers are filmed and recorded at each of the multiple remote locations, and the video and audio are transmitted to the event venue or other remote locations, and output from a large video display device or speaker at each location.

[0003] This two-way transmission of video and audio allows athletes (or performers) and spectators at the event venue, as well as viewers in multiple remote locations, to feel a sense of presence and unity as if they were in the same space (event venue) and experiencing the same thing, despite being physically far apart.

[0004] RTP (Real-time Transport Protocol) is often used for real-time transmission of video and audio over IP networks, but the data transmission time between two locations varies depending on the communication lines connecting the two locations. For example, consider the case where video and audio filmed and recorded at event venue A at time T is transmitted to two remote locations B and C, and then the video and audio filmed and recorded at remote locations B and C are transmitted back to event venue A. The video and audio filmed and recorded at event venue A at time T and transmitted to remote location B is transmitted back to remote location B at time T. b1 and at time T b1 The video and audio recorded at time T b2 At this time, in the remote location C, the video and audio that was shot, recorded, and transmitted at the event venue A at time T is played back at time T c1 (≠T b1 ) and played at remote location C at time T c1 The video and audio recorded at time T are transmitted back to event venue A, where they are c2 (≠T b2 ) may be played.

[0005] In such a case, for the athletes (or performers) and spectators at event venue A, the video and audio showing how viewers in multiple remote locations reacted to the event they experienced at time T are transmitted to each other at different times (time T b2 and time T c2 ) for the athletes (or performers) and spectators at event venue A, this can make it difficult to intuitively understand the connection to their own experience and can feel unnatural, making it difficult to increase the sense of unity with spectators in remote locations. Also, when the video and audio transmitted from event venue A and the video and audio transmitted from remote location B are played back in remote location C, the spectators in remote location C can feel the intuitive difficulty of understanding and unnaturalness mentioned above.

[0006] In order to eliminate such intuitive difficulty and unnaturalness, a method has been used in the past in which multiple videos and multiple audios transmitted from multiple remote locations are synchronized and played at event venue A. When synchronizing the playback timing of videos and audios, the sending and receiving sides are synchronized using NTP (Network Time Protocol) or PTP (Precision Time Protocol) so that both sides manage the same time information, and the video and audio data are packetized into RTP packets at the time of transmission. At this time, the absolute time at the moment the video and audio are sampled is added as an RTP timestamp, and the receiving side generally delays at least one of the videos and audios based on the time information to adjust the timing and achieve synchronization (Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Synchronous playback technology for audio signals distributed via IP networks (Tokumoto, Ikedo, Kaneko, Kataoka, IEICE Transactions on Computer Science and Engineering, Vol. J87-D-II No.9 pp.1870-1883) Summary of the Invention [Problem to be solved by the invention]

[0008] However, in conventional methods for synchronizing video and audio playback, the playback timing is adjusted to the video or audio with the greatest delay time, which causes the real-timeness of the video and audio playback timing to be lost, making it difficult to reduce the discomfort felt by viewers.In other words, it is necessary to devise ways to play back video and audio in a way that reduces the discomfort felt by viewers when playing multiple videos and audio transmitted at different times from multiple locations.

[0009] This invention has been made in light of the above-mentioned circumstances, and its purpose is to provide a technique for reducing the sense of discomfort felt by viewers when multiple video and audio streams transmitted at different times from multiple locations are played back. [Means for solving the problem]

[0010] In one embodiment of the present invention, a media processing device is an apparatus at a first location and includes a receiving unit that receives a packet storing second media acquired at the second location at a time when a first media acquired at the first location at a first time is to be played at the second location, and a processing unit that generates third media from the second media in accordance with a processing mode based on the second time and the first time associated with receiving the packet storing the second media, and outputs the third media to a presentation device. Effect of the Invention

[0011] According to one aspect of the present invention, it is possible to reduce the sense of incongruity felt by viewers when a plurality of video and audio streams transmitted at different times from a plurality of locations are played back. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of each electronic device included in the media processing system according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the software configuration of each electronic device constituting the media processing system according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of a data structure of a video time management DB provided in the server of the base R1 according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a data structure of a voice time management DB provided in the server of the base R1 according to the first embodiment. [Diagram 5]FIG. 5 is a flowchart showing the procedure and content of video processing by the server at the site O according to the first embodiment. [Figure 6] FIG. 6 is a flowchart showing the video processing procedure and processing contents of the server at the site R1 according to the first embodiment. [Figure 7] FIG. 7 is a flowchart showing the procedure and contents of a transmission process of an RTP packet storing a video Vsignal1 of a server at a site O according to the first embodiment. [Figure 8] FIG. 8 is a flowchart showing the procedure and contents of a reception process of an RTP packet storing a video Vsignal1 of a server at a site R1 according to the first embodiment. [Figure 9] FIG. 9 is a flowchart showing the procedure and content of the calculation process of the presented time t1 of the server at the site R1 according to the first embodiment. [Figure 10] FIG. 10 is a flowchart showing the procedure and contents of a transmission process of an RTP packet storing a video Vsignal2 of the server at the site R1 according to the first embodiment. [Figure 11] FIG. 11 is a flowchart showing the procedure and contents of a reception process of an RTP packet storing a video Vsignal2 of the server at the site O according to the first embodiment. [Figure 12] FIG. 12 is a flowchart showing the procedure and contents of the processing of the video Vsignal2 of the server at the site O according to the first embodiment. [Figure 13] FIG. 13 is a flowchart showing the procedure and content of voice processing by the server at the site O according to the first embodiment. [Figure 14] FIG. 14 is a flowchart showing the procedure and content of voice processing by the server at the site R1 according to the first embodiment. [Figure 15] FIG. 15 is a flowchart showing the procedure and contents of a transmission process of an RTP packet storing the audio Asignal1 of the server at the site O according to the first embodiment. [Figure 16]FIG. 16 is a flowchart showing the procedure and content of a reception process of an RTP packet storing the audio Asignal1 of the server at the site R1 according to the first embodiment. [Figure 17] FIG. 17 is a flowchart showing the procedure and content of a transmission process of an RTP packet storing the audio Asignal2 of the server at the site R1 according to the first embodiment. [Figure 18] FIG. 18 is a flowchart showing the procedure and contents of a reception process of an RTP packet storing the audio Asignal2 of the server at the site O according to the first embodiment. [Figure 19] FIG. 19 is a flowchart showing the procedure and content of processing of the voice Asignal2 by the server at the site O according to the first embodiment. [Figure 20] FIG. 20 is a block diagram showing an example of the hardware configuration of each electronic device included in the media processing system according to the second embodiment. [Figure 21] FIG. 21 is a block diagram showing an example of the software configuration of each electronic device constituting the media processing system according to the second embodiment. [Figure 22] FIG. 22 is a diagram showing an example of a data structure of a voice time management DB included in the server of the base R2 according to the second embodiment. [Figure 23] FIG. 23 is a flowchart showing the video processing procedure and processing contents of the server at the site R1 according to the second embodiment. [Figure 24] FIG. 24 is a flowchart showing the video processing procedure and processing contents of the server at the site R2 according to the second embodiment. [Diagram 25] FIG. 25 is a flowchart showing the procedure and content of processing of the video Vsignal2 of the server at the site R2 according to the second embodiment. [Figure 26] FIG. 26 is a flowchart showing the procedure and content of voice processing by the server at the location R1 according to the second embodiment. [Figure 27]FIG. 27 is a flowchart showing the procedure and content of voice processing by the server at the site R2 according to the second embodiment. [Figure 28] FIG. 28 is a flowchart showing the procedure and content of a reception process of an RTP packet storing the audio Asignal1 of the server at the site R2 according to the second embodiment. [Figure 29] FIG. 29 is a flowchart showing the procedure and content of a calculation process for the presented time t2 by the server at the site R2 according to the second embodiment. [Diagram 30] FIG. 30 is a flowchart showing the procedure and content of processing of the voice Asignal2 by the server at the site R2 according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, several embodiments of the present invention will be described with reference to the drawings. The time information that is uniquely determined with respect to the absolute time when the video and audio were shot and recorded at the site O, which is an event venue such as a competition venue or a concert venue, is transmitted to multiple remote sites R1 to R n (n is an integer of 2 or more) is assigned to the video and audio transmitted from point R1 to point R n In each of the above, the video and audio recorded at the time when the video and audio with the time information is played back is associated with the time information. n When playing back the video and audio transmitted from each of the above, the video and audio are processed based on the time information before being played back.

[0014] The time information is sent from site O to site R1. n The time information is transmitted and received between each of the points R1 to R2 by any of the following means. n Each of these is associated with the video and audio shot and recorded. (1) The time information is sent from site O to site R1. nThe time information is stored in the header extension area of ​​the RTP packet transmitted to and received from each of the above. For example, the time information is in absolute time format (hh:mm:ss.fff format), but it may also be in millisecond format. (2) The time information is sent from site O to site R1. n The time information is written using APP (Application-Defined) in RTCP (RTP Control Protocol), which is transmitted and received at regular intervals between each of the . In this example, the time information is in millisecond format. (3) At the start of transmission, the time information is n The time information is stored in a Session Description Protocol (SDP) that describes the initial value parameters to be exchanged between each of the devices. In this example, the time information is in millisecond format.

[0015] [First embodiment] In the first embodiment, at the site O, n In this embodiment, the video and audio transmitted back from the broadcasting station are processed and played back.

[0016] The time information used to process the video and audio is n The time information is stored in the header extension area of ​​the RTP packets transmitted to and from each of the above. For example, the time information is in absolute time format (hh:mm:ss.fff format).

[0017] Although the video and audio are described as being converted into RTP packets and transmitted / received, this is not limited to the above. Video and audio may be processed / managed by the same functional unit / DB (database). Video and audio may be stored in a single RTP packet and transmitted / received. Video and audio are examples of media.

[0018] (Configuration example) FIG. 1 is a block diagram showing an example of the hardware configuration of each electronic device included in a media processing system S according to the first embodiment. The media processing system S includes a plurality of electronic devices included in the site O, sites R1 to R n Each of the bases includes a plurality of electronic devices and a time distribution server 10. The electronic devices and the time distribution server 10 at each base can communicate with each other via an IP network.

[0019] The location O includes a server 1, an event video shooting device 101, a return video presentation device 102, an event audio recording device 103, and a return audio presentation device 104. The location O is an example of a first location.

[0020] The server 1 is an electronic device that controls each electronic device included in the site O. The server 1 is an example of a media processing device. The event video shooting device 101 is a device including a camera that shoots video of the location O. The event video shooting device 101 is an example of a video shooting device. The loopback image display device 102 is a n The loopback image presentation device 102 is an apparatus including a display that reproduces and displays the images that are looped back and transmitted from each of the above to the location O. For example, the display is a liquid crystal display. The loopback image presentation device 102 is an example of an image presentation device or a presentation device. The event sound recording device 103 is a device including a microphone that records sound at the location O. The event sound recording device 103 is an example of a sound recording device. The return voice presentation device 104 is a n The return voice presentation device 104 is a device including a speaker that reproduces and outputs the voice that is looped back and transmitted from each of the above to the location O. The return voice presentation device 104 is an example of a voice presentation device or a presentation device.

[0021] An example of the configuration of the server 1 will be described. The server 1 includes a control unit 11, a program storage unit 12, a data storage unit 13, a communication interface 14, and an input / output interface 15. The elements included in the server 1 are connected to each other via a bus.

[0022] The control unit 11 corresponds to the central part of the server 1. The control unit 11 includes a processor such as a central processing unit (CPU). The control unit 11 includes a read only memory (ROM) as a non-volatile memory area. The control unit 11 includes a random access memory (RAM) as a volatile memory area. The processor loads a program stored in the ROM or the program storage unit 12 into the RAM. The processor executes the program loaded into the RAM, whereby the control unit 11 realizes each of the functional units described below. The control unit 11 constitutes a computer.

[0023] The program storage unit 12 is configured with a non-volatile memory that can be written and read at any time, such as a hard disk drive (HDD) or a solid state drive (SSD) as a storage medium. The program storage unit 12 stores programs necessary for executing various control processes. For example, the program storage unit 12 stores programs that cause the server 1 to execute processes by each functional unit (described later) realized in the control unit 11. The program storage unit 12 is an example of storage.

[0024] The data storage unit 13 is configured with a non-volatile memory that can be written and read at any time, such as an HDD or an SSD, as a storage medium. The data storage unit 13 is an example of a storage or a storage unit.

[0025] The communication interface 14 includes various interfaces that communicatively connect the server 1 to other electronic devices using a communication protocol defined by an IP network.

[0026] The input / output interface 15 is an interface that enables communication between the server 1 and each of the event video shooting device 101, the loopback video presentation device 102, the event audio recording device 103, and the loopback audio presentation device 104. The input / output interface 15 may include an interface for wired communication or an interface for wireless communication.

[0027] The hardware configuration of the server 1 is not limited to the above-mentioned configuration. The server 1 allows the above-mentioned components to be omitted or changed, and new components to be added, as appropriate.

[0028] The site R1 includes a server 2, a video presentation device 201, an offset video shooting device 202, a return video shooting device 203, an audio presentation device 204, and a return audio recording device 205. The site R1 is an example of a second site different from the first site.

[0029] The server 2 is an electronic device that controls each electronic device included in the base R1. The image presentation device 201 is a device including a display that plays and displays the image transmitted from the location O to the location R1. The image presentation device 201 is an example of a presentation device. The offset video camera 202 is a device capable of recording the shooting time. The offset video camera 202 is a device including a camera installed so as to be able to shoot the entire video display area of ​​the video presentation device 201. The offset video camera 202 is an example of a video camera. The loopback video capture device 203 is a device including a camera that captures video of the site R1. For example, the loopback video capture device 203 captures video of the site R1 where the video presentation device 201 that plays and displays the video transmitted from the site O to the site R1 is installed. The loopback video capture device 203 is an example of a video capture device. The audio presentation device 204 is a device including a speaker that reproduces and outputs the audio transmitted from the location O to the location R1. The audio presentation device 204 is an example of a presentation device. The return voice recording device 205 is a device including a microphone for recording voice at the site R1. For example, the return voice recording device 205 records voice of the site R1 where the voice presentation device 204 that reproduces and outputs the voice transmitted from the site O to the site R1 is installed. The return voice recording device 205 is an example of a voice recording device.

[0030] An example of the configuration of the server 2 will be described. The server 2 includes a control unit 21, a program storage unit 22, a data storage unit 23, a communication interface 24, and an input / output interface 25. The elements included in the server 2 are connected to each other via a bus. The control unit 21 may be configured in the same manner as the control unit 11. The processor loads a program stored in the ROM or the program storage unit 22 into the RAM. The processor executes the program loaded into the RAM, causing the control unit 21 to realize each of the functional units described below. The control unit 21 constitutes a computer. Program storage unit 22 may be configured similarly to program storage unit 12 . Data store 23 may be configured similarly to data store 13 . The communication interface 24 may be configured similarly to the communication interface 14. The communication interface 14 includes various interfaces that communicatively connect the server 2 to other electronic devices. The input / output interface 25 may be configured similarly to the input / output interface 15. The input / output interface 25 enables communication between the server 2 and each of the video presentation device 201, the offset video shooting device 202, the loopback video shooting device 203, the audio presentation device 204, and the loopback audio recording device 205. The hardware configuration of the server 2 is not limited to the above-mentioned configuration. The server 2 allows the above-mentioned components to be omitted or changed, and new components to be added, as appropriate. In addition, base R2 to base R n The hardware configurations of the electronic devices included in each of the bases R1 and R2 are similar to those of the base R1 described above, and therefore the description thereof will be omitted.

[0031] The time distribution server 10 is an electronic device that manages a reference system clock. The reference system clock is absolute time.

[0032] FIG. 2 is a block diagram showing an example of the software configuration of each electronic device that constitutes the media processing system S according to the first embodiment.

[0033] The server 1 includes a time management unit 111, an event video transmission unit 112, a return video reception unit 113, a return video processing unit 114, an event audio transmission unit 115, a return audio reception unit 116, and a return audio processing unit 117. Each functional unit is realized by execution of a program by the control unit 11. It can also be said that each functional unit is included in the control unit 11 or the processor. Each functional unit can be interpreted as the control unit 11 or the processor.

[0034] The time management unit 111 performs time synchronization with the time distribution server 10 using a known protocol such as NTP or PTP, and manages the reference system clock. The time management unit 111 manages a reference system clock that is the same as the reference system clock managed by the server 2. The reference system clock managed by the time management unit 111 and the reference system clock managed by the server 2 are time-synchronized.

[0035] The event video transmission unit 112 transmits the video V output from the event video shooting device 101 via an IP network. signal1 The RTP packet containing n The video is sent to each server. signal1 is the absolute time T video This is the image captured on . signal1 Acquiring the video V signal1 This includes filming the footage. signal1 Acquiring the video V captured by the event video capture device 101 signal1 This includes sampling the video V signal1 The RTP packet containing video At time T video is video V at base O signal1 is the time when the image was acquired. video is the time information for processing the loopback video at the location O. signal1 is an example of the first image. videois an example of a first time. The RTP packet is an example of a packet. The event video transmitting unit 112 is an example of a transmitting unit.

[0036] The return video receiving unit 113 receives the video V signal2 The RTP packet containing n Each server receives the video. signal2 is a video V signal1 Base R1 ~ Base R n The video is captured at this location at the time it is played at any one of the locations. signal2 The acquisition of the image V signal2 This includes filming the footage. signal2 The acquisition of the image V captured by the image capture device 203 is signal2 This includes sampling the video V signal2 The RTP packet containing video It has been granted. signal2 is an example of the second video. Return video receiving unit 113 is an example of a receiving unit.

[0037] The folded image processing unit 114 converts the image V signal2 Video from V signal3 Generate the video V signal3 The image is then output to the image display device 102. signal3 is an example of the third image. The folded image processing unit 114 is an example of a processing unit.

[0038] The event audio transmission unit 115 receives the audio A output from the event audio recording device 103 via the IP network. signal1 The RTP packet containing n The audio is sent to each server. signal1 is the absolute time T audio Audio A was recorded. signal1 Acquiring the voice A signal1 This includes recording Audio A signal1Acquiring the voice A recorded by the event voice recording device 103 signal1 This involves sampling the audio A. signal1 The RTP packet containing audio At time T audio is voice A at base O signal1 is the time when the image was acquired. audio is time information for processing the return voice at location O. signal1 is an example of the first voice. audio is an example of a first time. The event audio transmitting unit 115 is an example of a transmitting unit.

[0039] The return voice receiving unit 116 receives the voice A via the IP network. signal2 The RTP packet containing n Received from each server. Audio A signal2 is Voice A signal1 Base R1 ~ Base R n This is the audio recorded at this location at the time it is played at any of the locations. signal2 The acquisition of the voice A is performed by the return voice recording device 205. signal2 This includes recording Audio A signal2 The acquisition of the voice A recorded by the return voice recording device 205 is signal2 This involves sampling the audio A. signal2 The RTP packet containing audio It is given. Audio A signal2 is an example of the second voice. The return voice receiving unit 116 is an example of a receiving unit.

[0040] The return voice processing unit 117 processes the voice A signal2 Audio A signal3 Generate voice A signal3 The speech output device 104 outputs the speech A. signal3 is an example of the third voice. The callback voice processing unit 117 is an example of a processing unit.

[0041] The server 2 comprises a time management unit 211, an event video receiving unit 212, a video offset calculation unit 213, a return video transmission unit 214, an event audio receiving unit 215, a return audio transmission unit 216, a video time management DB 231 and an audio time management DB 232. Each functional unit is realized by execution of a program by the control unit 21. It can also be said that each functional unit is provided by the control unit 21 or a processor. Each functional unit can be interpreted as the control unit 21 or a processor. The video time management DB 231 and the audio time management DB 232 are realized by the data storage unit 23.

[0042] The time management unit 211 performs time synchronization with the time distribution server 10 using a known protocol such as NTP or PTP, and manages the reference system clock. The time management unit 211 manages a reference system clock that is the same as the reference system clock managed by the server 1. The reference system clock managed by the time management unit 211 and the reference system clock managed by the server 1 are time-synchronized.

[0043] The event video receiving unit 212 receives the video V signal1 The event video receiving unit 212 receives an RTP packet storing the video V from the server 1. signal1 is output to the image presentation device 201. The image offset calculation unit 213 calculates the image V signal1 Then, a presentation time t1 is calculated, which is the absolute time when the content is reproduced. The return video transmission unit 214 returns the video V signal2 The RTP packet containing the video V is sent to Server 1. signal2 The RTP packet containing the video V signal2 The time T associated with the presentation time t1, which coincides with the absolute time t at which the image was taken. video Includes.

[0044] The event voice receiving unit 215 receives voice A via an IP network. signal1 The event voice receiving unit 215 receives an RTP packet storing the voice A from the server 1. signal1is output to the audio presentation device 204. The return voice transmission unit 216 transmits the voice A via the IP network. signal2 The RTP packet containing the above is sent to Server 1. signal2 The RTP packet containing audio Includes.

[0045] FIG. 3 is a diagram showing an example of a data structure of the video time management DB 231 provided in the server 2 of the base R1 according to the first embodiment. The video time management DB 231 stores the time T video and presentation time t1 in association with each other. The video time management DB 231 includes a video synchronization reference time column and a presentation time column. video The presentation time column stores the presentation time t1.

[0046] FIG. 4 is a diagram showing an example of a data structure of the voice time management DB 232 provided in the server 2 of the base R1 according to the first embodiment. The voice time management DB 232 stores the time T audio and Audio A signal1 It is a database that stores and associates the above. The audio time management DB 232 includes an audio synchronization reference time column and an audio data column. audio The voice data column stores voice A signal1 Store.

[0047] In addition, base R2 to base R n Each server of the base R2 to the base R3 includes the same functional unit and DB as the server 1 of the base R1, and executes the same processes as the server 1 of the base R1. n The processing flow and DB structure of the functional parts included in each server will not be described.

[0048] (Example of operation) The following will be described taking the operations of the site O and the site R1 as an example.n The operation of the base R1 may be the same as that of the base R1, and the description thereof will be omitted. n It may also be read as follows.

[0049] (1) Processing and playback of loopback video The video processing of the server 1 in the location O will be described. FIG. 5 is a flowchart showing the procedure and contents of video processing by the server 1 at the site O according to the first embodiment. The event video transmission unit 112 transmits the video V signal1 The RTP packet storing the above information is transmitted to the server 2 at the site R1 (step S11). A typical example of the process at step S11 will be described later. The return video receiving unit 113 receives the video V signal2 The RTP packet storing the above is received from the server 2 at the site R1 (step S12). A typical example of the process at step S12 will be described later. The loopback image processing unit 114 processes the image V by the loopback image receiving unit 113. signal2 The current time T n and time T video Based on the processing method, the image V signal2 Video from V signal3 The folded image processing unit 114 generates the image V signal3 The converted image is then output to the image presentation device 102 (step S13). A typical example of the process in step S13 will be described later.

[0050] The video processing of the server 2 at the site R1 will be described. FIG. 6 is a flowchart showing the procedure and content of video processing by the server 2 at the site R1 according to the first embodiment. The event video receiving unit 212 receives the video V signal1 The RTP packet storing the above information is received from the server 1 (step S14). A typical example of the process in step S14 will be described later. The image offset calculation unit 213 calculates the image V signal1 A typical example of the process in step S15 will be described later. The return video transmission unit 214 returns the video V signal2 The RTP packet storing the above information is transmitted to the server 1 (step S16). A typical example of the process of step S16 will be described later.

[0051] The following describes typical examples of the processes of steps S11 to S13 of the above-mentioned server 1 and steps S14 to S16 of the above-mentioned server 2. In order to describe the processes in chronological order, the processes will be described in the order of step S11 of the server 1, step S14 of the server 2, step S15 of the server 2, step S16 of the server 2, step S12 of the server 1, and step S13 of the server 1.

[0052] FIG. 7 shows the video V of the server 1 at the site O according to the first embodiment. signal1 7 is a flowchart showing the procedure and processing contents of a transmission process of an RTP packet storing the above-mentioned data. The event video transmission unit 112 receives the video V output from the event video shooting device 101. signal1 At regular intervals I video (step S111). The event video transmission unit 112 transmits the video signal1 In step S112, for example, the event video transmission unit 112 generates an RTP packet storing the acquired video V signal1 The event video transmission unit 112 stores the video V in an RTP packet based on the reference system clock managed by the time management unit 111. signal1 The absolute time T video The event video transmission unit 112 receives the obtained time T video is stored in the header extension area of ​​the RTP packet. The event video transmission unit 112 transmits the generated video Vsignal1 The RTP packet storing the above is sent to the IP network (step S113).

[0053] FIG. 8 shows the video V of the server 2 at the site R1 according to the first embodiment. signal1 8 is a flowchart showing a procedure and process contents of a reception process of an RTP packet storing the above-mentioned information. The event video receiving unit 212 receives the video V transmitted from the event video transmitting unit 112 via the IP network. signal1 The RTP packet storing the above is received (step S141). The event video receiving unit 212 receives the video V signal1 The video V stored in the RTP packet signal1 (step S142). The event video receiving unit 212 receives the acquired video V signal1 The image display device 201 outputs the image V signal1 Play and display. The event video receiving unit 212 receives the video V signal1 The time T stored in the header extension area of ​​the RTP packet video is obtained (step S144). The event video receiving unit 212 receives the acquired video V signal1 and time T video to image offset calculation section 213 (step S145).

[0054] 9 is a flowchart showing the procedure and details of the calculation process of the presented time t1 by the server 2 at the base R1 according to the first embodiment. The image offset calculation unit 213 calculates the image V signal1 and time T video is acquired from the event video receiving unit 212 (step S151). The image offset calculation unit 213 calculates the offset of the acquired image V signal1and calculates the presentation time t1 based on the video input from the offset video shooting device 202 (step S152). In step S152, for example, the video offset calculation unit 213 calculates the presentation time t1 based on the video input from the offset video shooting device 202 using a known image processing technique. signal1 The video offset calculation unit 213 extracts a video frame including the above-mentioned video offset. The video offset calculation unit 213 acquires the shooting time given to the extracted video frame as the presentation time t1. The shooting time is an absolute time. The video offset calculation unit 213 calculates the obtained time T video is stored in the video synchronization reference time column of the video time management DB 231 (step S153). The video offset calculation unit 213 stores the acquired presentation time t1 in the presentation time column of the video time management DB 231 (step S154).

[0055] FIG. 10 shows the video V of the server 2 at the site R1 according to the first embodiment. signal2 10 is a flowchart showing a typical example of the process of step S16 of the server 2. The process of step S16 of the server 2 is shown in FIG. The loopback video transmission unit 214 receives the video V output from the loopback video shooting device 203. signal2 At regular intervals I video (Step S161). signal2 The image display device 201 displays the image V signal1 This is video captured at location R1 at the time it is played back at location R1. The return video transmission unit 214 transmits the acquired video V signal2 In step S162, for example, the return video transmission unit 214 calculates a time t that is an absolute time when the video V signal2 The time code T indicates the time of shooting. c If an absolute time is given, t = T c The time t is obtained as the image V signal2 Time code T c If no time is added, the return video transmission unit 214 calculates the current time Tn The return video transmission unit 214 obtains a predetermined value t video_offset (positive number) t = T n -t video_offset to obtain the time t.

[0056] The return video transmission unit 214 refers to the video time management DB 231 and extracts a record having a time t1 that matches the acquired time t (step S163). The return video transmission unit 214 refers to the video time management DB 231 and extracts the time T video is obtained (step S164). The return video transmission unit 214 receives the video V signal2 In step S165, for example, the return video transmission unit 214 generates an RTP packet storing the acquired video V signal2 The return video transmission unit 214 stores the obtained time T video is stored in the header extension area of ​​the RTP packet. The return video transmission unit 214 transmits the generated video V signal2 The RTP packet storing the above is sent to the IP network (step S166).

[0057] FIG. 11 shows the video V of the server 1 at the site O according to the first embodiment. signal2 11 is a flowchart showing a typical example of the process of step S12 of the server 1. The process of step S12 of the server 1 is shown in FIG. The return video receiving unit 113 receives the video V transmitted from the return video transmitting unit 214 via the IP network. signal2 The RTP packet storing the above is received (step S121). The return video receiving unit 113 receives the video V signal2 The video V stored in the RTP packet signal2 (Step S122). The return video receiving unit 113 receives the video V signal2The time T stored in the header extension area of ​​the RTP packet video (step S123). The return video receiving unit 113 receives the acquired video V signal2 and time T video The image is then passed back to the image processing unit 114 (step S124).

[0058] FIG. 12 shows the video V of the server 1 at the site O according to the first embodiment. signal2 12 is a flowchart showing the processing procedure and processing contents of the above. The folded image processing unit 114 converts the image V signal2 and time T video is obtained from the return video receiving unit 113 (step S131). The loopback image processing unit 114 calculates the current time T n (Step S132). The current time T n The return video receiving unit 113 receives the video V signal2 The current time T n is a video V signal2 It can also be said that the current time T n is a video V signal2 Image V generated based on signal3 It can also be said that the playback time of the video V signal2 The current time T n is an example of the second time.

[0059] The loopback image processing unit 114 processes the acquired current time T n and time T video The captured image V is processed according to the processing state based on signal2 Video from V signal3 In step S133, for example, the folded image processing unit 114 generates a current time T n and time T video The difference betweenn - T video )(ms) based on the value of the image V signal2 The wraparound image processing unit 114 determines the processing mode of (T n - T video ) based on the value of the image V signal2 The aliased image processing unit 114 changes the processing mode so as to lower the quality of the image as the difference value increases. signal2 Processing and video V signal2 The processing mode may include not performing any processing on the image V. signal2 The folded image processing unit 114 processes the image V signal2 When processing the image V signal3 is video V signal2 The loopback image processing unit 114 processes the image V signal2 If no processing is done on the image V signal3 is video V signal2 is the same as:

[0060] The folded image processing unit 114 processes the folded image so that the visibility is reduced when the folded image is played back on the folded image display device 102. signal2 The image is then replayed on the video display device 102 so that the viewer does not feel uncomfortable (T n - T video If the value of V is small, the aliased image processing unit 114 signal2 No processing is performed on (T n - T video Even if the value of V) is too large, the folded image processing unit 114 processes the image V so that the image does not become completely unrecognizable. signal2 For example, the image V signal2 The processing to change the display size of the image V will be described. signal2 If the horizontal pixels are w and the vertical pixels are h, the image V generated according to the processing mode is signal3 The horizontal pixel w' and vertical pixel h' are as follows: (1) 0ms ≤ T n - T videoWhen ≦ 300 ms w' = w, h' = h (2) 300ms < T n - T video When ≦ 500 ms w' = {-(1 / 400)( T n - T video ) + 7 / 4}*w, h' = {-(1 / 400)( T n - T video ) + 7 / 4} * h (3) 500ms < T n - T video When w' = 0.5 * w, h' = 0.5 * h The processing is not limited to the above as a change in image quality, and may be, in addition to the above-mentioned change in display size, blurring the image with a Gaussian filter, reducing the brightness of the image, etc. signal3 Video V signal2 Other processing may be used as long as it reduces visibility more than the above.

[0061] The folded image processing unit 114 converts the generated image V signal3 The loopback image presentation device 102 outputs the video V signal2 Based on the video V signal3 Play and display.

[0062] (2) Processing and playback of loopback audio The voice processing of the server 1 at the site O will be described. FIG. 13 is a flowchart showing the procedure and contents of voice processing by the server 1 at the site O according to the first embodiment. The event voice transmission unit 115 transmits the voice A via the IP network. signal1 The RTP packet storing the above information is transmitted to the server 2 at the site R1 (step S17). A typical example of the process at step S17 will be described later. The return voice receiving unit 116 receives the voice A via the IP network.signal2 The RTP packet storing the above is received from the server 2 at the site R1 (step S18). A typical example of the process at step S18 will be described later. The return voice processing unit 117 receives the voice A from the return voice receiving unit 116. signal2 The current time T n and time T audio Depending on the processing state based on the voice A signal2 Audio A signal3 The return voice processing unit 117 generates the voice A signal3 The speech output unit 104 outputs the result to the speech output unit 104 (step S19). A typical example of the process in step S19 will be described later.

[0063] The voice processing of the server 2 at the site R1 will be described. FIG. 14 is a flowchart showing the procedure and contents of voice processing by the server 2 at the base R1 according to the first embodiment. The event voice receiving unit 215 receives voice A via an IP network. signal1 The RTP packet storing the above information is received from the server 1 (step S20). A typical example of the process of step S20 will be described later. The return voice transmission unit 216 transmits the voice A via the IP network. signal2 The RTP packet storing the above information is transmitted to the server 1 (step S21). A typical example of the process of step S21 will be described later.

[0064] The following describes typical examples of the processes of steps S17 to S19 of the above-mentioned server 1 and steps S20 to S21 of the above-mentioned server 2. In order to describe the processes in chronological order, the processes will be described in the order of step S17 of the server 1, step S20 of the server 2, step S21 of the server 2, step S18 of the server 1, and step S19 of the server 1.

[0065] FIG. 15 shows the voice A of the server 1 at the site O according to the first embodiment. signal115 is a flowchart showing a typical example of the process of step S17 of the server 1. The process of step S17 of the server 1 is shown in FIG. The event voice transmission unit 115 receives the voice A output from the event voice recording device 103. signal1 At regular intervals I audio (step S171). The event voice transmission unit 115 transmits voice A signal1 In step S172, for example, the event voice transmission unit 115 generates an RTP packet storing the acquired voice A. signal1 The event voice transmission unit 115 stores the acquired voice A in an RTP packet based on the reference system clock managed by the time management unit 111. signal1 The absolute time T audio The event voice transmission unit 115 acquires the acquired time T audio is stored in the header extension area of ​​the RTP packet. The event voice transmission unit 115 generates the voice A signal1 The RTP packet storing the above is sent to the IP network (step S173).

[0066] FIG. 16 shows the voice A of the server 2 at the site R1 according to the first embodiment. signal1 16 is a flowchart showing a procedure and process contents of a reception process of an RTP packet storing the above-mentioned information. The event voice receiving unit 215 receives the voice A sent from the event voice transmitting unit 115 via the IP network. signal1 The RTP packet storing the above is received (step S201). The event voice receiving unit 215 receives the voice A signal1 Audio A stored in the RTP packet signal1 (Step S202). The event voice receiving unit 215 receives the acquired voice A signal1 The voice presentation device 204 outputs the voice A to the voice presentation device 204 (step S203). signal1 Play and output. The event voice receiving unit 215 receives the voice A signal1 The time T stored in the header extension area of ​​the RTP packet audio (Step S204). The event voice receiving unit 215 receives the acquired voice A signal1 and time T audio In step S205, for example, the event voice receiving unit 215 stores the acquired time T audio The event voice receiving unit 215 stores the acquired voice A in the voice synchronization reference time column of the voice time management DB 232. signal1 is stored in the voice data column of the voice time management DB 232.

[0067] FIG. 17 shows the voice A of the server 2 at the site R1 according to the first embodiment. signal2 17 is a flowchart showing a typical example of the process of step S21 of the server 2. The process of step S21 of the server 2 is shown in FIG. The return voice transmission unit 216 receives the voice A output from the return voice recording device 205. signal2 At regular intervals I audio (Step S211). signal2 The voice presentation device 204 outputs voice A signal1 This is the audio captured at location R1 at the time it is played back at location R1. The return voice transmission unit 216 refers to the voice time management DB 232 and signal2 The voice A acquired by the return voice transmission unit 216 is extracted (step S212). signal2 is the audio A reproduced by the audio presentation device 204. signal1 and a voice generated at the location R1 (such as cheers from the audience at the location R1). In step S212, for example, the return voice transmission unit 216 separates the two voices using a known voice analysis technique. The return voice transmission unit 216 separates the two voices into voice A and voice B reproduced by the voice presentation device 204 by the separation of the voices. signal1The return voice transmission unit 216 refers to the voice time management DB 232 and identifies the voice A reproduced by the identified voice presentation device 204. signal1 The return voice transmission unit 216 searches for voice data that matches the specified voice A reproduced by the specified voice presentation device 204 by referring to the voice time management DB 232. signal1 Extract records that have audio data that matches. The return voice transmission unit 216 refers to the voice time management DB 232 and extracts the time T audio (step S213). The return voice transmission unit 216 receives the voice A signal2 In step S214, for example, the return voice transmission unit 216 generates an RTP packet storing the acquired voice A. signal2 The return voice transmission unit 216 stores the obtained time T audio is stored in the header extension area of ​​the RTP packet. The return voice transmission unit 216 returns the generated voice A signal2 The RTP packet storing the above is sent to the IP network (step S215).

[0068] FIG. 18 shows the voice A of the server 1 at the site O according to the first embodiment. signal2 18 is a flowchart showing a procedure and process contents of a receiving process of an RTP packet storing the above-mentioned information. The return voice receiving unit 116 receives the voice A sent from the return voice transmitting unit 216 via the IP network. signal2 The RTP packet storing the above is received (step S181). The return voice receiving unit 116 receives the voice A signal2 Audio A stored in the RTP packet signal2 is obtained (step S182). The return voice receiving unit 116 receives the voice A signal2 The time T stored in the header extension area of ​​the RTP packet audiois obtained (step S183). The return voice receiving unit 116 receives the acquired voice A signal2 and time T audio The voice is returned to the voice processing unit 117 (step S184).

[0069] FIG. 19 shows a voice A of the server 1 at the site O according to the first embodiment. signal2 19 is a flowchart showing the processing procedure and processing contents of the above. The return voice processing unit 117 processes the voice A signal2 and time T audio is obtained from the return voice receiving unit 116 (step S191). The return voice processing unit 117 calculates the current time T n (Step S192). The current time T n The return voice receiving unit 116 receives the voice A signal2 The current time T n is Voice A signal2 It can also be said that the current time T n is Voice A signal2 Speech A generated based on signal3 This can also be called the playback time of Audio A. signal2 The current time T n is an example of the second time.

[0070] The return voice processing unit 117 receives the current time T n and time T audio According to the processing state based on signal2 Audio A signal3 In step S193, for example, the return voice processing unit 117 generates a current time T n and time T audio The difference between n - T audio )(ms)signal2 The return voice processing unit 117 determines the processing mode of (T n - T audio ) based on the value of Voice A signal2 The return voice processing unit 117 changes the processing mode so as to lower the quality of the voice as the difference value increases. signal2 Processing and audio A signal2 The processing mode may include both processing and not processing the voice A. signal2 The degree of processing for the voice A is included. signal2 When processing audio A signal3 is Audio A signal2 The return voice processing unit 117 is different from the voice A. signal2 If no processing is applied to audio A, signal3 is Audio A signal2 is the same as:

[0071] The return voice processing unit 117 processes the sound so that the sound is less audible when it is played back by the return voice presentation device 104. signal2 The audio is then played back by the audio presentation device 104 so that the viewer does not feel uncomfortable (T n - T audio If the value of (a) is small, the return voice processing unit 117 signal2 No processing is performed on (T n - T audio Even if the value of A) is too large, the return voice processing unit 117 adjusts the return voice so that the voice does not become completely inaudible. signal2 For example, audio A signal2 The processing to change the strength of the voice A will be described. signal2 If the strength of is s, the voice A generated according to the processing mode is signal3 The strength s' of is: (1) 0ms ≤ T n - T audio When ≦ 100ms, s' = s (2) 100ms < T n - Taudio When ≦ 300ms, s' = {- (1 / 400)( T n - T audio ) + 5 / 4} * s (3) 300ms < T n - T audio When s' = 0.5 * s The processing is not limited to the above as a change in the quality of the voice, and may be, in addition to the above-mentioned change in sound intensity, (T n - T audio The higher the value of (T n - T audio The larger the value of )(ms), the further away the sound seems to come. signal3 is Audio A signal2 Other processing may be used as long as the processing reduces the audibility more than the above.

[0072] The return voice processing unit 117 generates the voice A signal3 The return speech presentation device 104 outputs the speech A to be returned from the point R1 to the point O (step S194). signal2 Based on Voice A signal3 Play and output.

[0073] (effect) As described above, in the first embodiment, the server 1 n and time T video Based on the processing method, the image V signal2 Video from V signal3 In a typical example, server 1 generates a n and time T videoThe processing mode is changed based on the difference value between the two. The server 1 may change the processing mode so as to lower the quality of the image as the difference value increases. In this way, the server 1 can process the image so that the image becomes less noticeable when played back. Generally, when viewing an image projected onto a screen or the like from a certain point X, the image can be clearly seen as long as the distance from point X to the screen is within a certain range. On the other hand, as the distance increases, the image becomes smaller and blurrier, making it difficult to see.

[0074] Server 1 is at the current time T n and time T audio Depending on the processing state based on the voice A signal2 Audio A signal3 In a typical example, server 1 generates a n and time T audio The processing mode is changed based on the difference between the original sound and the original sound. The server 1 may change the processing mode so as to lower the quality of the sound as the difference becomes larger. In this way, the server 1 can process the sound so that the sound becomes difficult to hear when played back. Generally, when listening to sound played back from a speaker or the like from a certain point X, if the distance from point X to the speaker (sound source) is within a certain range, the sound can be clearly heard at the same time as the sound source is generated. On the other hand, as the distance increases, the sound travels with a delay from the time of sound playback and is attenuated, making it difficult to hear.

[0075] Server 1 is at the current time T n and time T video Or the current time T n and time T audio By performing processing to reproduce the above-mentioned viewing experience based on the above, it is possible to convey the state of a viewer at a physically distant location while reducing the sense of discomfort caused by large data transmission delay times.

[0076] In this way, the server 1 can reduce the sense of discomfort felt by the viewer when a plurality of video and audio streams transmitted at different times from a plurality of locations are played back.

[0077] [Second embodiment] The second embodiment is an embodiment in which, when video and audio transmitted from location O and video and audio transmitted from multiple remote locations other than location R are played back at a remote location R, the video and audio transmitted from the multiple remote locations other than location R are processed and played back.

[0078] The time information used to process the video and audio is n The time information is stored in the header extension area of ​​the RTP packets transmitted to and from each of the above. For example, the time information is in absolute time format (hh:mm:ss.fff format).

[0079] The following explanation focuses on two remote locations, location R1 and location R2, and describes the process of playing back the video and audio transmitted from location O and the video and audio transmitted from location R1 at location R2. Explanations will be omitted for the reception process at location O of the video and audio transmitted back from locations R1 and R2, the reception process and processing process at location R1 of the video and audio transmitted from location R2, and the transmission process at location R2 of the video and audio filmed and recorded at location R2 to locations O and R1.

[0080] Although the video and audio are described as being converted into RTP packets and transmitted, this is not a limitation. Video and audio may be processed and managed by the same functional unit / DB (database). Video and audio may both be stored in a single RTP packet and transmitted.

[0081] (Configuration example) In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted. In the second embodiment, the differences from the first embodiment will be mainly described.

[0082] FIG. 20 is a block diagram showing an example of the hardware configuration of each electronic device included in the media processing system S according to the second embodiment. The media processing system S includes a plurality of electronic devices included in the site O, sites R1 to R n Each of the bases includes a plurality of electronic devices and a time distribution server 10. The electronic devices and the time distribution server 10 at each base can communicate with each other via an IP network. Similar to the first embodiment, the location O includes the server 1, the event video shooting device 101, and the event audio recording device 103. The location O is an example of a first location.

[0083] Similar to the first embodiment, the site R1 includes a server 2, an image presentation device 201, an offset image shooting device 202, and an audio presentation device 204. Unlike the first embodiment, the site R1 includes an image shooting device 206 and an audio recording device 207. The site R1 is an example of a second site. The video imaging device 206 is a device including a camera that captures video of the site R1. For example, the video imaging device 206 captures video of the site R1 where the video presentation device 201 that plays and displays video transmitted from the site O to the site R1 is installed. The video imaging device 206 is an example of a video imaging device. The voice recording device 207 is a device including a microphone for recording voice at the site R1. For example, the voice recording device 207 records voice of the situation at the site R1 where the voice presentation device 204 that reproduces and outputs the voice transmitted from the site O to the site R1 is installed. The voice recording device 207 is an example of a voice recording device.

[0084] The site R2 includes a server 3, a video presentation device 301, an offset video shooting device 302, an audio presentation device 303, and an offset audio recording device 304. The site R2 is an example of a third site different from the first site and the second site. The server 3 is an electronic device that controls each electronic device included in the site R2. The server 3 is an example of a media processing device. The image presentation device 301 displays the image transmitted from the site O to the site R2, as well as the images from the sites R1 and R3 to R nThe video presentation device 301 is an example of a presentation device. The offset video camera 302 is a device capable of recording the shooting time. The offset video camera 302 is a device including a camera installed so as to be able to shoot the entire video display area of ​​the video presentation device 301. The offset video camera 302 is an example of a video camera. The audio presentation device 303 receives audio transmitted from the site O to the site R2, and audio transmitted from the sites R1 and R3 to R n The audio presentation device 303 is an example of a presentation device. The offset audio recording device 304 is a device capable of recording the recording time. The offset audio recording device 304 is a device including a microphone installed so as to record the audio played back by the audio presentation device 303. The offset audio recording device 304 is an example of an audio recording device.

[0085] An example of the configuration of the server 3 will be described. The server 3 includes a control unit 31, a program storage unit 32, a data storage unit 33, a communication interface 34, and an input / output interface 35. The elements included in the server 3 are connected to each other via a bus. The control unit 31 may be configured in the same manner as the control unit 11. The processor loads a program stored in the ROM or the program storage unit 32 into the RAM. The processor executes the program loaded into the RAM, causing the control unit 31 to realize each of the functional units described below. The control unit 31 constitutes a computer. Program storage unit 32 may be configured similarly to program storage unit 12 . Data store 33 may be configured similarly to data store 13 . The communication interface 34 may be configured similarly to the communication interface 14. The communication interface 34 includes various interfaces that communicatively connect the server 3 to other electronic devices. The input / output interface 35 may be configured similarly to the input / output interface 15. The input / output interface 35 enables communication between the server 3 and each of the video presentation device 301, the offset video shooting device 302, the audio presentation device 303, and the offset audio recording device 304. The hardware configuration of the server 3 is not limited to the above-mentioned configuration. The server 3 allows the above-mentioned components to be omitted or changed, and new components to be added, as appropriate.

[0086] FIG. 21 is a block diagram showing an example of the software configuration of each electronic device constituting the media processing system S according to the second embodiment.

[0087] As in the first embodiment, the server 1 includes a time management unit 111, an event video transmission unit 112, and an event audio transmission unit 115. Each functional unit is realized by execution of a program by the control unit 11. It can also be said that each functional unit is included in the control unit 11 or the processor. Each functional unit can be interpreted as the control unit 11 or the processor.

[0088] Similar to the first embodiment, the server 2 includes a time management unit 211, an event video receiving unit 212, a video offset calculation unit 213, an event audio receiving unit 215, a video time management DB 231, and an audio time management DB 232. Unlike the first embodiment, the server 2 includes a video transmitting unit 217 and an audio transmitting unit 218. Each functional unit is realized by execution of a program by the control unit 21. It can also be said that each functional unit is included in the control unit 21 or the processor. Each functional unit can be read as the control unit 21 or the processor. The video time management DB 231 and the audio time management DB 232 are realized by the data storage unit 23.

[0089] The video transmission unit 217 transmits the video V signal2 The RTP packet containing the video V is sent to server 3. signal2 The RTP packet containing the video V signal2The time T associated with the presentation time t1, which coincides with the absolute time t at which the image was taken. video Including. Video V signal2 is an example of the second image. An RTP packet is an example of a packet. video is an example of a first time.

[0090] The voice transmission unit 218 transmits the voice A via the IP network. signal2 The RTP packet containing the above is sent to Server 3. signal2 The RTP packet containing audio Includes: Audio A signal2 is an example of the second voice. audio is an example of a first time.

[0091] The server 3 comprises a time management unit 311, an event video receiving unit 312, a video offset calculation unit 313, a video receiving unit 314, a video processing unit 315, an event audio receiving unit 316, an audio offset calculation unit 317, an audio receiving unit 318, an audio processing unit 319, a video time management DB 331 and an audio time management DB 332. Each functional unit is realized by execution of a program by the control unit 31. It can also be said that each functional unit is provided by the control unit 31 or the processor. Each functional unit can be interpreted as the control unit 31 or the processor. The video time management DB 331 and the audio time management DB 332 are realized by the data storage unit 33.

[0092] The time management unit 311 performs time synchronization with the time distribution server 10 using a known protocol such as NTP or PTP, and manages the reference system clock. The time management unit 311 manages the same reference system clock as the reference system clock managed by the servers 1 and 2. The reference system clock managed by the time management unit 311 and the reference system clock managed by the servers 1 and 2 are time-synchronized.

[0093] The event video receiving unit 312 receives the video V signal1 The event video receiving unit 312 receives an RTP packet storing the video V from the server 1.signal1 The event video receiving unit 312 outputs the video to the video presentation device 301. The event video receiving unit 312 is an example of a first receiving unit. signal1 is an example of the first image. The image offset calculation unit 313 calculates the image V on the image presentation device 301 at the site R2. signal1 The video offset calculation unit 313 calculates a presentation time t1, which is the absolute time when the video is played back. The video offset calculation unit 313 is an example of a calculation unit. The presentation time t1 is an example of a third time. The video receiving unit 314 receives the video V signal2 The RTP packet containing the n The video receiving unit 314 receives the video from each of the servers. The video receiving unit 314 is an example of a second receiving unit. The image processing unit 315 processes the image V signal2 Video from V signal3 Generate the video V signal3 The image processing unit 315 outputs the image V to the image display device 301. signal3 is an example of the third image.

[0094] The event voice receiving unit 316 receives voice A via an IP network. signal1 The event voice receiving unit 316 receives an RTP packet storing the voice A from the server 1. signal1 The event voice receiving unit 316 outputs the voice A to the voice presentation device 303. The event voice receiving unit 316 is an example of a first receiving unit. signal1 is an example of the first voice. The audio offset calculation unit 317 calculates the audio offset of the audio A by the audio presentation device 303 at the site R2. signal1 The audio offset calculation unit 317 calculates a presentation time t2, which is the absolute time when the audio offset is reproduced. The audio offset calculation unit 317 is an example of a calculation unit. The presentation time t2 is an example of a third time. The voice receiving unit 318 receives voice A via the IP network. signal2 The RTP packet containing the n The voice receiving unit 318 receives the voice from each of the servers. The voice receiving unit 318 is an example of a second receiving unit. The audio processing unit 319 processes audio A signal2 Audio Asignal3 Generate voice A signal3 The voice processing unit 319 outputs the voice A to the voice presentation device 303. The voice processing unit 319 is an example of a processing unit. signal3 is an example of a third voice.

[0095] The video time management DB 331 may have the same data structure as the video time management DB 231. video and presentation time t1 in association with each other. The video time management DB 331 is an example of a storage unit.

[0096] FIG. 22 is a diagram showing an example of a data structure of the voice time management DB 332 included in the server 3 of the base R2 according to the second embodiment. The audio time management DB 332 stores the time T audio and presentation time t2 in association with each other. The audio time management DB 332 is an example of a storage unit. The audio time management DB 332 includes an audio synchronization reference time column and a presentation time column. audio The presentation time column stores the presentation time t2.

[0097] (Example of operation) In the following, the operations of the sites O, R1, and R2 will be described as examples.

[0098] (1) Video processing and playback The video processing of the server 1 in the location O will be described. The event video transmission unit 112 transmits the video V signal1 The RTP packet containing n The video is sent to each server. signal1 The RTP packet containing video At time T video is each site other than site O (R1, R2, ..., R n) is time information for processing the video. The process of the event video transmission unit 112 may be the same as the process described in the first embodiment with reference to FIG. 7, and the description thereof will be omitted.

[0099] The video processing of the server 2 at the site R1 will be described. FIG. 23 is a flowchart showing the procedure and content of video processing by the server 2 at the site R1 according to the second embodiment. The event video receiving unit 212 receives the video V signal1 The RTP packet storing the above is received from the server 1 (step S22). A typical example of the process of the event video receiving unit 212 in step S22 may be similar to the process described in the first embodiment with reference to FIG. 8, and therefore a description thereof will be omitted.

[0100] The image offset calculation unit 213 calculates the image V signal1 The presentation time t1 at which the content is reproduced is calculated (step S23). A typical example of the process of the image offset calculation unit 213 in step S23 may be similar to the process described in the first embodiment with reference to FIG. 9, and the description thereof will be omitted.

[0101] The video transmission unit 217 transmits the video V signal2 The RTP packet storing the above is transmitted to the server 3 (step S24). A typical example of the process of the video transmission unit 217 in step S24 may be similar to the process of the return video transmission unit 214 described in the first embodiment with reference to FIG. In the description using Figure 10, the notations "return video imaging device 203" and "return video transmission unit 214" are replaced with "video imaging device 206" and "video transmission unit 217", and the description of the processing of the video transmission unit 217 is omitted.

[0102] The video processing by the server 3 at the site R2 will be described. FIG. 24 is a flowchart showing the video processing procedure and processing contents of the server 3 at the site R2 according to the second embodiment. The event video receiving unit 312 receives the video V signal1 The RTP packet storing the above is received from the server 1 (step S25). A typical example of the process of the event video receiving unit 312 in step S25 may be similar to the process of the event video receiving unit 212 described in the first embodiment with reference to FIG. In the description using Figure 8, the terms "video presentation device 201", "event video receiving unit 212", and "video offset calculation unit 213" are replaced with "video presentation device 301", "event video receiving unit 312", and "video offset calculation unit 313", thereby omitting explanation of the processing of event video receiving unit 312.

[0103] The image offset calculation unit 313 calculates the image V signal1 The presentation time t1 at which the content is reproduced is calculated (step S26). A typical example of the process of the image offset calculation unit 313 in step S26 may be similar to the process of the image offset calculation unit 213 described in the first embodiment with reference to FIG. In the description using Figure 9, the terms "offset video shooting device 202", "event video receiving unit 212", "video offset calculation unit 213", and "video time management DB 231" are replaced with "offset video shooting device 302", "event video receiving unit 312", "video offset calculation unit 313", and "video time management DB 331", thereby omitting explanation of the processing of video offset calculation unit 313.

[0104] The video receiving unit 314 receives the video V signal2 The RTP packet storing the above is received from the server 2 at the site R1 (step S27). A typical example of the process of the video receiving unit 314 in step S27 may be similar to the process of the return video receiving unit 113 described in the first embodiment with reference to FIG. In the description using Figure 11, the terms "return video receiving unit 113", "return video processing unit 114" and "return video transmitting unit 214" are replaced with "video transmitting unit 217", "video receiving unit 314" and "video processing unit 315", and the description of the processing of video receiving unit 314 will be omitted.

[0105] The image processing unit 315 processes the image V by the image receiving unit 314. signal2 The current time T n and the image V according to the processing state based on the presentation time t1. signal2 Video from V signal3 The image processing unit 315 generates the image V signal3 to the image presentation device 301 (step S28).

[0106] FIG. 25 shows the video V of the server 3 at the site R2 according to the second embodiment. signal2 25 is a flowchart showing the processing procedure and processing contents of the above. The image processing unit 315 processes the image V signal2 and time T video is acquired from the video receiving unit 314 (step S281). The video processing unit 315 refers to the video time management DB 331 and video Then, a record having a video synchronization reference time that matches the video synchronization reference time is extracted (step S282). The video processing unit 315 refers to the video time management DB 331, and acquires the presentation time t1 from the presentation time column of the extracted record (step S283).

[0107] The image processing unit 315 calculates the current time T n (Step S284). The current time T n The video receiving unit 314 receives the video V signal2 The current time T n is a video V signal2 It can also be said that the current time Tn is a video V signal2 Image V generated based on signal3 It can also be said that the playback time of the video V signal2 The current time T n is an example of the second time.

[0108] The image processing unit 315 processes the acquired current time T n According to the processing state based on the presentation time t1, the acquired image V signal2 Video from V signal3 In step S285, for example, the image processing unit 315 generates the current time T n and the difference between presentation time t1, that is, (T n - Based on the value of t1) (ms), the image V signal2 The image processing unit 315 determines the processing mode of the image (T n - t1) based on the value of the image V signal2 The image processing unit 315 changes the processing mode so as to lower the quality of the image as the difference value increases. signal2 Processing and video V signal2 The processing mode may include not performing any processing on the image V. signal2 This includes the degree of processing.

[0109] The image processing unit 315 processes the image so that the image is less visible when played back on the image display device 301. signal2 The image is reproduced on the image display device 301 so that the viewer does not feel uncomfortable (T n - t1) is small, the image processing unit 315 signal2 No processing is performed on (T n Even if the value of t1) is too large, the image processing unit 315 processes the image V so that the image does not become completely unrecognizable. signal2 For example, the image V signal2 The processing to change the display size of the image V will be described. signal2If the horizontal pixels are w and the vertical pixels are h, the image V generated according to the processing mode is signal3 The horizontal pixel w' and vertical pixel h' are as follows: (1) 0ms ≤ T n - When t1≦ 300 ms w' = w, h' = h (2) 300ms < T n - When t1≦ 500 ms w' = {- (1 / 400)( T n - t1) + 7 / 4}*w, h' = {- (1 / 400)( T n - t1) + 7 / 4} * h (3) 500ms < T n - At t1 w' = 0.5 * w, h' = 0.5 * h The processing is not limited to the above as a change in image quality, and may be, in addition to the above-mentioned change in display size, blurring the image with a Gaussian filter, reducing the brightness of the image, etc. signal3 Video V signal2 Other processing may be used as long as the processing reduces visibility more than the above.

[0110] The image processing unit 315 processes the generated image V signal3 The image display device 301 outputs the image to the image display device 301 (step S286). n The video V transmitted from each of these to site R2 signal2 Based on the video V signal3 Play and display.

[0111] (2) Audio processing and playback The voice processing of the server 1 at the site O will be described. The event voice transmission unit 115 transmits the voice A via the IP network. signal1 The RTP packet containing n The audio is sent to each server. signal1 The RTP packet containingaudio At time T audio is each site other than site O (R1, R2, ..., R n ) is time information for processing the voice. The process of the event voice transmission unit 115 may be the same as the process described in the first embodiment with reference to FIG. 15, and the description thereof will be omitted.

[0112] The voice processing of the server 2 at the site R1 will be described. FIG. 26 is a flowchart showing the procedure and details of voice processing by the server 2 at the base R1 according to the second embodiment. The event voice receiving unit 215 receives voice A via an IP network. signal1 The RTP packet storing the above is received from the server 1 (step S29). A typical example of the process of the event voice receiving unit 215 in step S29 may be similar to the process described in the first embodiment with reference to FIG. 16, and therefore a description thereof will be omitted.

[0113] The voice transmission unit 218 transmits the voice A via the IP network. signal2 The RTP packet storing the above is transmitted to the server 3 (step S30). A typical example of the process of the voice transmitting unit 218 in step S30 may be similar to the process of the return voice transmitting unit 216 described in the first embodiment with reference to FIG. In the description using Figure 17, the notations "callback voice recording device 205" and "callback voice transmission unit 216" are replaced with "voice recording device 207" and "voice transmission unit 218", and the description of the processing of the voice transmission unit 218 is omitted.

[0114] The voice processing of the server 3 at the site R2 will be described. FIG. 27 is a flowchart showing the procedure and contents of voice processing by the server 3 at the site R2 according to the second embodiment. The event voice receiving unit 316 receives voice A via an IP network. signal1The RTP packet storing the above is received from the server 1 (step S31). A typical example of the process of step S31 will be described later.

[0115] The audio offset calculation unit 317 calculates the offset of the audio A from the audio presentation device 303. signal1 A typical example of the process of step S32 will be described later.

[0116] The voice receiving unit 318 receives voice A via the IP network. signal2 The RTP packet storing the above is received from the server 2 at the site R1 (step S33). A typical example of the process of the voice receiving unit 318 in step S33 may be similar to the process of the return voice receiving unit 116 described in the first embodiment with reference to FIG. In the description using Figure 18, the terms "callback voice receiving unit 116", "callback voice processing unit 117", and "callback voice transmitting unit 216" are replaced with "voice receiving unit 318", "voice processing unit 319", and "voice transmitting unit 218", and the description of the processing of the voice receiving unit 318 will be omitted.

[0117] The voice processing unit 319 receives the voice A from the voice receiving unit 318. signal2 The current time T n According to the processing mode based on the presentation time t2, voice A signal2 Audio A signal3 The voice processing unit 319 generates the voice A signal3 to the audio presentation device 303 (step S34). A typical example of the process in step S34 will be described later.

[0118] FIG. 28 shows the voice A of the server 3 at the site R2 according to the second embodiment. signal1 28 is a flowchart showing a procedure and process contents of receiving an RTP packet storing the above-mentioned information. The event voice receiving unit 316 receives the voice A sent from the event voice transmitting unit 115 via the IP network.signal1 The RTP packet storing the above is received (step S311). The event voice receiving unit 316 receives the voice A signal1 Audio A stored in the RTP packet signal1 (Step S312). The event voice receiving unit 316 receives the acquired voice A signal1 The voice presentation device 303 outputs the voice A to the voice presentation device 303 (step S313). signal1 Play and output. The event voice receiving unit 316 receives the voice A signal1 The time T stored in the header extension area of ​​the RTP packet audio (step S314). The event voice receiving unit 316 receives the acquired voice A signal1 and time T audio to the audio offset calculation unit 317 (step S315).

[0119] 29 is a flowchart showing the procedure and details of the calculation process of the presented time t2 by the server 3 at the base R2 according to the second embodiment. The audio offset calculation unit 317 calculates the audio offset signal1 and time T audio is acquired from the event voice receiving unit 316 (step S321). The audio offset calculation unit 317 calculates the offset of the acquired audio A. signal1 Based on the voice input from the offset voice recording device 304, the presentation time t2 is calculated (step S322). signal1 The audio offset calculation unit 317 separates the two audio signals by using a known audio analysis technique, for example. The audio offset calculation unit 317 separates the audio signals into audio A and audio B, and outputs the audio A and audio B to the audio presentation device 303. signal1 Then, the presentation time t2, which is the absolute time when the content is played back, is acquired. The audio offset calculation unit 317 calculates the acquired time T audio The audio synchronization reference time is stored in the audio time management DB 332 (step S323). The audio offset calculation unit 317 stores the acquired presentation time t2 in the presentation time column of the audio time management DB 332 (step S324).

[0120] FIG. 30 shows the voice A of the server 3 at the site R2 according to the second embodiment. signal2 30 is a flowchart showing the processing procedure and processing contents of the above. The audio processing unit 319 processes audio A signal2 and time T audio is acquired from the voice receiving unit 318 (step S341). The voice processing unit 319 refers to the voice time management DB 332 and obtains the time T audio Then, a record having an audio synchronization reference time that matches the audio synchronization reference time is extracted (step S342). The voice processing unit 319 refers to the voice time management DB 332 and acquires the presentation time t2 from the presentation time column of the extracted record (step S343).

[0121] The voice processing unit 319 calculates the current time T n (Step S344). The current time T n The voice receiving unit 318 receives voice A. signal2 The current time T n is Voice A signal2 It can also be said that the current time T n is Voice A signal2 Speech A generated based on signal3 This can also be called the playback time of Audio A. signal2 The current time T n is an example of the second time.

[0122] The voice processing unit 319 receives the current time Tn According to the processing mode based on the presentation time t2, the acquired voice A signal2 Audio A signal3 In step S345, for example, the voice processing unit 319 generates the current time T n and the difference between presentation time t2, that is, (T n - Based on the value of t2) (ms), signal2 The audio processing unit 319 determines the processing mode of (T n - Based on the value of t2) signal2 The audio processing unit 319 changes the processing mode so as to lower the quality of the audio as the difference value increases. signal2 Processing and audio A signal2 The processing mode may include both processing and not processing the voice A. signal2 This includes the degree of processing.

[0123] The audio processing unit 319 processes the audio so that the audibility is reduced when the audio is played back by the audio presentation device 303. signal2 The audio presentation device 303 can be used to play the audio so that the viewer does not feel uncomfortable (T n If the value of t2) is small, the voice processing unit 319 converts the voice A signal2 No processing is performed on (T n Even if the value of t2) is too large, the voice processing unit 319 processes the voice A so that the voice does not become completely inaudible. signal2 For example, audio A signal2 The processing to change the strength of the voice A will be described. signal2 If the strength of is s, the voice A generated according to the processing mode is signal3 The strength s' of is: (1) 0ms ≤ T n - When t2≦ 100ms, s' = s (2) 100ms < T n - When t2≦ 300ms, s' = {- (1 / 400)( T n - t2) + 5 / 4} * s (3) 300ms < T n - t2 s' = 0.5 * s The processing is not limited to the above as a change in the quality of the voice, and may be, in addition to the above-mentioned change in sound intensity, (T n The higher the value of t2) (ms), the lower the threshold value becomes, and the higher the frequency components are reduced by low-pass filtering. n - The larger the value of t2) (ms), the further away the sound seems to come. signal3 is Audio A signal2 Other processing may be used as long as the processing reduces the audibility more than the above.

[0124] The voice processing unit 319 processes the generated voice A signal3 The voice presentation device 303 outputs the voice to the voice presentation device 303 (step S346). n Audio A is transmitted from each of these to site R2. signal2 Based on Voice A signal3 Play and output.

[0125] (effect) As described above, in the second embodiment, the server 3 n and the image V according to the processing state based on the presentation time t1. signal2 Video from V signal3 In a typical example, the server 3 generates the current time T n and presentation time t1. The server 3 may change the processing mode so as to lower the quality of the image as the difference value increases. In this way, the server 3 can process the image so that the image becomes less noticeable when played back. Generally, when viewing an image projected onto a screen or the like from a certain point X, the image can be clearly viewed as long as the distance from point X to the screen is within a certain range. On the other hand, as the distance increases, the image becomes smaller and blurrier, making it difficult to view.

[0126] Server 3 is at the current time T nAccording to the processing mode based on the presentation time t2, voice A signal2 Audio A signal3 In a typical example, the server 3 generates the current time T n and presentation time t2. The server 3 may change the processing mode so as to lower the quality of the voice as the difference value increases. In this way, the server 3 can process the voice so that the voice becomes difficult to hear when played back. In general, when listening to voice played back from a speaker or the like from a certain point X, if the distance from point X to the speaker (sound source) is within a certain range, the voice can be clearly heard at the same time as the sound source is generated. On the other hand, as the distance increases, the sound travels with a delay from the playback time of the sound and is attenuated, making it difficult to hear.

[0127] Server 3 is at the current time T n and presentation time t1 or current time T n By performing processing to reproduce the above-mentioned viewing experience based on the presentation time t2, it is possible to convey the situation of viewers at physically distant locations while reducing the sense of discomfort caused by large data transmission delay times.

[0128] In this way, the server 3 can reduce the sense of discomfort felt by the viewer when a plurality of video and audio streams transmitted at different times from a plurality of locations are played back.

[0129] [Other embodiments] The media processing device may be realized as a single device as described in the above example, or may be realized as multiple devices with distributed functions.

[0130] The program may be transferred in a state where it is stored in an electronic device, or in a state where it is not stored in an electronic device. In the latter case, the program may be transferred via a network, or in a state where it is recorded on a recording medium. The recording medium is a non-transitory tangible medium. The recording medium is a computer-readable medium. The recording medium may be in any form, such as a CD-ROM or a memory card, as long as it is capable of storing the program and is readable by a computer.

[0131] Although the embodiment of the present invention has been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, in implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted.

[0132] In short, this invention is not limited to the above-mentioned embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining multiple components disclosed in the above-mentioned embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0133] 1 Server 2 Server 3 Server 10. Time distribution server 11 Control section 12 Program memory section 13 Data storage unit 14 Communication Interface 15 Input / Output Interface 21 Control section 22 Program memory section 23 Data storage unit 24 Communication Interface 25 Input / Output Interface 31 Control Unit 32 Program memory section 33 Data storage unit 34 Communication Interface 35 Input / Output Interface 101 Event video recording equipment 102 Reflected image display device 103 Event audio recording device 104 Return voice presentation device 111 Time Management Department 112 Event video transmission unit 113 Return video receiving unit 114 Return Video Processing Department 115 Event audio transmission unit 116 Return voice receiving unit 117 Return voice processing section 201 Image display device 202 Offset video equipment 203 Loopback video camera 204 Audio presentation device 205 Call-back voice recording device 206 Video equipment 207 Audio Recording Device 211 Time Management Department 212 Event video receiving unit 213 Video offset calculation unit 214 Return video transmission unit 215 Event audio receiver 216 Return voice transmission unit 217 Video transmission unit 218 Voice transmission unit 231 Video time management DB 232 Audio time management DB 301 Video display device 302 Offset video equipment 303 Audio presentation device 304 Offset Audio Recording Device 311 Time Management Department 312 Event video receiving unit 313 Video offset calculation unit 314 Video Receiving Unit 315 Video Processing Department 316 Event audio receiver 317 Audio offset calculation unit 318 Voice Receiving Unit 319 Audio Processing Department 331 Video time management DB 332 Audio time management DB O Base R1~R n Location S Media Processing System

Claims

1. a media processing device at a first location, a receiving unit that receives a packet storing a second media acquired at the second location at a time when the first media acquired at the first location at a first time is to be played at the second location; a processing unit that generates a third media from the second media in accordance with a processing mode based on a second time associated with receiving a packet storing the second media and the first time, and outputs the third media to a presentation device; Equipped with The processing mode includes changing the quality of at least one of video and audio of the second media. Media processing equipment.

2. The media processing device according to claim 1 , wherein the processing section changes the processing mode based on a difference between the second time and the first time.

3. a media processing device at a third location different from the first location and the second location, a first receiving unit that receives a packet storing a first media acquired at a first time in the first location and outputs the first media to a presentation device; a second receiving unit that receives a packet storing a second media acquired at the second location at a time when the first media is to be played back at the second location; a processing unit that generates a third media from the second media in accordance with a processing mode based on a second time associated with receiving a packet storing the second media and a third time at which the first media is played on the presentation device, and outputs the third media to the presentation device; A media processing device comprising:

4. The media processing device according to claim 3 , wherein the processing section changes the processing mode based on a difference between the second time and the third time.

5. 5. The media processing device according to claim 2, wherein the processing section changes the processing mode so as to degrade the quality of the media as the difference value increases.

6. 1. A media processing method using a media processing device at a first location, comprising: receiving a packet storing a second media acquired at the second location at a time when the first media acquired at the first location at a first time is to be played at the second location; generating a third media from the second media in accordance with a processing mode based on a second time associated with receiving a packet storing the second media and the first time; outputting the third media to a presentation device; and Equipped with The processing mode includes changing the quality of at least one of video and audio of the second media. Media processing methods.

7. A media processing method using a media processing device at a third location different from the first location and the second location, comprising: receiving a packet storing a first media acquired at a first time at the first location; outputting the first media to a presentation device; receiving a packet storing a second media acquired at the second location at a time when the first media is to be played at the second location; generating a third media from the second media in accordance with a processing state based on a second time associated with receiving a packet storing the second media and a third time at which the first media is played on the presentation device; outputting the third media to the presentation device; and A media processing method comprising:

8. A media processing program for causing a computer to execute processing by each unit of the media processing device according to any one of claims 1 to 5.

Images