Media processing device, media processing method and media processing program

The media processing device synchronizes playback across multiple locations by adjusting for transmission delays, addressing the loss of real-time timing in conventional methods to enhance viewer coherence.

JP7586326B2Active Publication Date: 2024-11-19NIPPON TELEGRAPH & TELEPHONE CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023532955
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-07-07
Publication Date
2024-11-19
Estimated Expiration
2041-07-07

AI Technical Summary

Technical Problem

Conventional video/audio playback synchronization methods using RTP and NTP/PTP lose the real-time nature of playback timing, causing discomfort due to differing transmission delays across multiple locations, making it difficult to maintain a sense of unity and coherence among viewers.

Method used

A media processing device and method that adjusts playback timing by incorporating transmission delay information into RTP packets, allowing for synchronized playback across multiple locations by processing and managing video and audio data based on absolute time and transmission delays.

Benefits of technology

Reduces viewer discomfort by ensuring synchronized playback across multiple locations, maintaining a coherent viewing experience despite varying transmission delays.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586326000001
    Figure 0007586326000001
  • Figure 0007586326000002
    Figure 0007586326000002
  • Figure 0007586326000003
    Figure 0007586326000003
Patent Text Reader

Abstract

According to an embodiment of the present invention, a media processing device in a second site different from a first site includes: a first reception unit that receives, from an electronic device in the first site, a notification regarding a transmission delay time that is based on a first time when a medium is obtained at the first site and a second time associated with the reception, by the electronic device in the first site, of a packet related to a medium obtained at the second site at a time when the medium obtained at the first site is played at the second site; a second reception unit that receives, from the electronic device in the first site, a packet containing a first medium obtained at the first site and outputs the first medium to a presentation device; a processing unit that generates, from a second medium obtained at the second site at a time when the first medium is played at the second site, a third medium according to a processing manner based on the transmission delay time; and a transmission unit that transmits the third medium to the electronic device in the first site.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] One aspect of the present invention relates to a media processing device, a media processing method, and a media processing program. [Background technology]

[0002] In recent years, video and audio playback devices have come into use, which digitize video and audio filmed and recorded at a certain location, transmit them to a remote location in real time via communication lines such as an IP (Internet Protocol) network, and play the video and audio at the remote location. For example, public viewing, in which video and audio of a sports competition held at a sports venue or video and audio of a music concert held at a concert venue are transmitted to a remote location in real time, is being actively carried out. Such video and audio transmission is not limited to one-to-one one-way transmission. Bidirectional transmission is also performed, in which video and audio are transmitted from a venue where a sports competition is held (hereinafter referred to as an event venue) to multiple remote locations, and the video and audio of spectators enjoying the event and the audio such as cheers are filmed and recorded at each of the multiple remote locations, and the video and audio are transmitted to the event venue or other remote locations, and output from a large video display device or speaker at each location.

[0003] This two-way transmission of video and audio allows athletes (or performers) and spectators at the event venue, as well as viewers in multiple remote locations, to feel a sense of presence and unity as if they were in the same space (event venue) and experiencing the same thing, despite being physically far apart.

[0004] RTP (Real-time Transport Protocol) is often used for real-time transmission of video and audio over IP networks, but the data transmission time between two locations varies depending on the communication lines connecting the two locations. For example, consider the case where video and audio filmed and recorded at event venue A at time T is transmitted to two remote locations B and C, and then the video and audio filmed and recorded at remote locations B and C are transmitted back to event venue A. The video and audio filmed and recorded at event venue A at time T and transmitted to remote location B is transmitted back to remote location B at time T. b1 and at time T b1 The video and audio recorded at time T b2 At this time, in the remote location C, the video and audio that was shot, recorded, and transmitted at the event venue A at time T is played back at time T c1 (≠T b1 ) and played at remote location C at time T c1 The video and audio recorded at time T are transmitted back to event venue A, where they are c2 (≠T b2 ) may be played.

[0005] In such a case, for the athletes (or performers) and spectators at event venue A, the video and audio showing how viewers in multiple remote locations reacted to the event they experienced at time T are transmitted to each other at different times (time T b2 and time T c2 ) for the athletes (or performers) and spectators at event venue A, this can make it difficult to intuitively understand the connection to their own experience and can feel unnatural, making it difficult to increase the sense of unity with spectators in remote locations. Also, when the video and audio transmitted from event venue A and the video and audio transmitted from remote location B are played back in remote location C, the spectators in remote location C can feel the intuitive difficulty of understanding and unnaturalness mentioned above.

[0006] In order to eliminate such intuitive difficulty and unnaturalness, a method has been used in the past in which multiple videos and multiple audios transmitted from multiple remote locations are synchronized and played at event venue A. When synchronizing the playback timing of videos and audios, the sending and receiving sides are synchronized using NTP (Network Time Protocol) or PTP (Precision Time Protocol) so that both sides manage the same time information, and the video and audio data are packetized into RTP packets at the time of transmission. At this time, the absolute time at the moment the video and audio are sampled is added as an RTP timestamp, and the receiving side generally delays at least one of the videos and audios based on the time information to adjust the timing and achieve synchronization (Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Synchronous playback technology for audio signals distributed via IP networks (Tokumoto, Ikedo, Kaneko, Kataoka, IEICE Transactions on Computer Science and Engineering, Vol. J87-D-II No.9 pp.1870-1883) Summary of the Invention [Problem to be solved by the invention]

[0008] However, in conventional methods for synchronizing video and audio playback, the playback timing is adjusted to the video or audio with the longest delay time, which causes the real-time playback timing of the video and audio to be lost, making it difficult to reduce the discomfort felt by viewers. In other words, it is necessary to devise a way to play back video and audio in a way that reduces the discomfort felt by viewers when playing multiple videos and audio transmitted at different times from multiple locations. It is also necessary to shorten the data transmission time for video and audio transmitted from multiple locations.

[0009] This invention has been made in light of the above-mentioned circumstances, and its purpose is to provide a technique for reducing the sense of discomfort felt by viewers when multiple video and audio streams transmitted at different times from multiple locations are played back. [Means for solving the problem]

[0010] In one embodiment of the present invention, a media processing device is a media processing device at a second location different from a first location, and includes a first receiving unit that receives from an electronic device at the first location a notification regarding a transmission delay time based on a first time when media was acquired at the first location and a second time associated with the electronic device at the first location receiving a packet related to media acquired at the second location at the time when the media is to be played at the second location, a second receiving unit that receives from the electronic device at the first location a packet storing the first media acquired at the first location and outputs the first media to a presentation device, a processing unit that generates a third media from the second media acquired at the second location at the time when the first media is to be played at the second location in accordance with a processing mode based on the transmission delay time, and a transmitting unit that transmits the third media to the electronic device at the first location. Effect of the Invention

[0011] According to one aspect of the present invention, it is possible to reduce the sense of incongruity felt by viewers when a plurality of video and audio streams transmitted at different times from a plurality of locations are played back. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of each electronic device included in the media processing system according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the software configuration of each electronic device constituting the media processing system according to the first embodiment. [Diagram 3]FIG. 3 is a diagram showing an example of a data structure of a video time management DB provided in the server of the base R1 according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a data structure of a voice time management DB provided in the server of the base R1 according to the first embodiment. [Diagram 5] FIG. 5 is a flowchart showing the procedure and content of video processing by the server at the site O according to the first embodiment. [Figure 6] FIG. 6 is a flowchart showing the video processing procedure and processing contents of the server at the site R1 according to the first embodiment. [Figure 7] FIG. 7 is a flowchart showing the procedure and contents of a transmission process of an RTP packet storing a video Vsignal1 of a server at a site O according to the first embodiment. [Figure 8] FIG. 8 is a flowchart showing the procedure and contents of a reception process of an RTP packet storing a video Vsignal1 of a server at a site R1 according to the first embodiment. [Figure 9] FIG. 9 is a flowchart showing the procedure and content of the calculation process of the presented time t1 of the server at the site R1 according to the first embodiment. [Figure 10] FIG. 10 is a flowchart showing the procedure and contents of a reception process of an RTP packet storing a video Vsignal3 of the server at the site O according to the first embodiment. [Figure 11] FIG. 11 is a flowchart showing the procedure and processing content of a transmission process of an RTCP packet storing Δdx_video of the server at the site O according to the first embodiment. [Figure 12] FIG. 12 is a flowchart showing the procedure and content of a reception process of an RTCP packet storing Δdx_video by the server at the site R1 according to the first embodiment. [Figure 13] FIG. 13 is a flowchart showing the procedure and contents of the processing of the video Vsignal2 of the server at the site R1 according to the first embodiment. [Figure 14]FIG. 14 is a flowchart showing the procedure and contents of a transmission process of an RTP packet storing a video Vsignal3 of the server at the site R1 according to the first embodiment. [Figure 15] FIG. 15 is a flowchart showing the procedure and content of voice processing by the server at the site O according to the first embodiment. [Figure 16] FIG. 16 is a flowchart showing the procedure and content of voice processing by the server at the site R1 according to the first embodiment. [Figure 17] FIG. 17 is a flowchart showing the procedure and contents of a transmission process of an RTP packet storing the audio Asignal1 of the server at the site O according to the first embodiment. [Figure 18] FIG. 18 is a flowchart showing the procedure and contents of a reception process of an RTP packet storing the audio Asignal1 of the server at the site R1 according to the first embodiment. [Figure 19] FIG. 19 is a flowchart showing the procedure and contents of a reception process of an RTP packet storing an audio Asignal3 of the server at the site O according to the first embodiment. [Figure 20] FIG. 20 is a flowchart showing the procedure and processing content of a transmission process of an RTCP packet storing Δdx_audio of the server at the site O according to the first embodiment. [Figure 21] FIG. 21 is a flowchart showing the procedure and content of a reception process of an RTCP packet storing Δdx_audio by the server at the site R1 according to the first embodiment. [Figure 22] FIG. 22 is a flowchart showing the procedure and content of processing of the voice Asignal2 by the server at the site R1 according to the first embodiment. [Diagram 23] FIG. 23 is a flowchart showing the procedure and content of a transmission process of an RTP packet storing the audio Asignal3 of the server at the site R1 according to the first embodiment. [Figure 24]FIG. 24 is a block diagram showing an example of the hardware configuration of each electronic device included in the media processing system according to the second embodiment. [Diagram 25] FIG. 25 is a block diagram showing an example of the software configuration of each electronic device constituting the media processing system according to the second embodiment. [Figure 26] FIG. 26 is a diagram showing an example of a data structure of a voice time management DB provided in the server of the base R2 according to the second embodiment. [Figure 27] FIG. 27 is a flowchart showing the video processing procedure and processing contents of the server at the site R1 according to the second embodiment. [Figure 28] FIG. 28 is a flowchart showing the video processing procedure and processing contents of the server at the site R2 according to the second embodiment. [Figure 29] FIG. 29 is a flowchart showing the procedure and processing content of a transmission process of an RTCP packet storing Δdx_video of the server at the site R2 according to the second embodiment. [Diagram 30] FIG. 30 is a flowchart showing the procedure and content of voice processing by the server at the site R1 according to the second embodiment. [Diagram 31] FIG. 31 is a flowchart showing the procedure and content of voice processing by the server at the site R2 according to the second embodiment. [Diagram 32] FIG. 32 is a flowchart showing the procedure and content of a reception process of an RTP packet storing the audio Asignal1 of the server at the site R2 according to the second embodiment. [Diagram 33] FIG. 33 is a flowchart showing the procedure and content of a calculation process for the presented time t2 by the server at the site R2 according to the second embodiment. [Diagram 34] FIG. 34 is a flowchart showing the procedure and processing content of a transmission process of an RTCP packet storing Δdx_video of the server at the site R2 according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, several embodiments of the present invention will be described with reference to the drawings. The time information that is uniquely determined with respect to the absolute time when the video and audio were shot and recorded at the site O, which is an event venue such as a competition venue or a concert venue, is transmitted to multiple remote sites R. 1 ~Base R n (n is an integer of 2 or more) is assigned to the video and audio transmitted to site R. 1 ~Base R n In each of the above, the video and audio recorded at the time when the video and audio with the time information was played back is processed based on the time information and the data transmission time between the destination points. The processed video and audio are transmitted to point O or another point R.

[0014] The time information is 1 ~Base R n The time information is transmitted and received between each of the sites R and R by any of the following means. 1 ~Base R n Each of these is associated with the video and audio shot and recorded. (1) Time information is provided by site O and site R. 1 ~Base R n The time information is stored in the header extension area of ​​the RTP packet transmitted to and received from each of the above. For example, the time information is in absolute time format (hh:mm:ss.fff format), but it may also be in millisecond format. (2) Time information is provided by site O and site R. 1 ~Base R n The time information is written using APP (Application-Defined) in RTCP (RTP Control Protocol), which is transmitted and received at regular intervals between each of the . In this example, the time information is in millisecond format. (3) The time information is transmitted between site O and site R at the start of transmission. 1 ~Base R n The time information is stored in a Session Description Protocol (SDP) that describes the initial value parameters to be exchanged between each of the devices. In this example, the time information is in millisecond format.

[0015] [First embodiment] In the first embodiment, the site O is 1 ~Base R n This embodiment reproduces video and audio transmitted back from the

[0016] The time information used to process the video and audio is 1 ~Base R n The time information is stored in the header extension area of ​​the RTP packet transmitted to and from each of the above. For example, the time information is in absolute time format (hh:mm:ss.fff format). The RTP packet is an example of a packet.

[0017] Although the video and audio are described as being converted into RTP packets and transmitted / received, this is not limited to the above. Video and audio may be processed and managed by the same functional unit / DB (database). Video and audio may be stored in a single RTP packet and transmitted / received. Video and audio are examples of media.

[0018] (Configuration example) FIG. 1 is a block diagram showing an example of the hardware configuration of each electronic device included in a media processing system S according to the first embodiment. The media processing system S is a system for processing multiple electronic devices at site O and multiple electronic devices at site R. 1 ~Base R n Each of the bases includes a plurality of electronic devices and a time distribution server 10. The electronic devices and the time distribution server 10 at each base can communicate with each other via an IP network.

[0019] The location O includes a server 1, an event video shooting device 101, a return video presentation device 102, an event audio recording device 103, and a return audio presentation device 104. The location O is an example of a first location.

[0020] The server 1 is an electronic device that controls each electronic device included in the site O. The event video shooting device 101 is a device including a camera that shoots video of the location O. The event video shooting device 101 is an example of a video shooting device. The return image display device 102 is located at the site R. 1 ~Base R n The loopback image presentation device 102 is an apparatus including a display that reproduces and displays the images that are looped back and transmitted from each of the above to the location O. For example, the display is a liquid crystal display. The loopback image presentation device 102 is an example of an image presentation device or a presentation device. The event sound recording device 103 is a device including a microphone that records sound at the location O. The event sound recording device 103 is an example of a sound recording device. The callback voice presentation device 104 is 1 ~Base R n The return voice presentation device 104 is a device including a speaker that reproduces and outputs the voice that is looped back and transmitted from each of the above to the location O. The return voice presentation device 104 is an example of a voice presentation device or a presentation device.

[0021] An example of the configuration of the server 1 will be described. The server 1 includes a control unit 11, a program storage unit 12, a data storage unit 13, a communication interface 14, and an input / output interface 15. The elements included in the server 1 are connected to each other via a bus.

[0022] The control unit 11 corresponds to the central part of the server 1. The control unit 11 includes a processor such as a central processing unit (CPU). The control unit 11 includes a read only memory (ROM) as a non-volatile memory area. The control unit 11 includes a random access memory (RAM) as a volatile memory area. The processor loads a program stored in the ROM or the program storage unit 12 into the RAM. The processor executes the program loaded into the RAM, whereby the control unit 11 realizes each of the functional units described below. The control unit 11 constitutes a computer.

[0023] The program storage unit 12 is configured with a non-volatile memory that can be written and read at any time, such as a hard disk drive (HDD) or a solid state drive (SSD) as a storage medium. The program storage unit 12 stores programs necessary for executing various control processes. For example, the program storage unit 12 stores programs that cause the server 1 to execute processes by each functional unit (described later) realized in the control unit 11. The program storage unit 12 is an example of storage.

[0024] The data storage unit 13 is configured with a non-volatile memory that can be written and read at any time, such as an HDD or an SSD, as a storage medium. The data storage unit 13 is an example of a storage or a storage unit.

[0025] The communication interface 14 includes various interfaces that communicatively connect the server 1 to other electronic devices using a communication protocol defined by an IP network.

[0026] The input / output interface 15 is an interface that enables communication between the server 1 and each of the event video shooting device 101, the loopback video presentation device 102, the event audio recording device 103, and the loopback audio presentation device 104. The input / output interface 15 may include an interface for wired communication or an interface for wireless communication.

[0027] The hardware configuration of the server 1 is not limited to the above-mentioned configuration. The server 1 allows the above-mentioned components to be omitted or changed, and new components to be added, as appropriate.

[0028] Base R 1 The site R includes a server 2, a video presentation device 201, an offset video shooting device 202, a loopback video shooting device 203, an audio presentation device 204, and a loopback audio recording device 205. 1 is an example of a second location that is different from the first location.

[0029] Server 2 is base R1 The server 2 is an electronic device that controls each of the electronic devices included in the server 2. The server 2 is an example of a media processing device. The image display device 201 is a display device for displaying images from a location O to a location R. 1 The video presentation device 201 is an example of a presentation device. The offset video camera 202 is a device capable of recording the shooting time. The offset video camera 202 is a device including a camera installed so as to be able to shoot the entire video display area of ​​the video presentation device 201. The offset video camera 202 is an example of a video camera. The return video camera 203 is located at the base R 1 For example, the loopback video capture device 203 is a device including a camera that captures video from point O to point R. 1 A video presentation device 201 is installed at a site R that plays and displays the video transmitted to the 1 The folded image capturing device 203 is an example of an image capturing device. The voice presentation device 204 is a device for 1 The audio presentation device 204 is an example of a presentation device. The return voice recording device 205 is 1 For example, the return voice recording device 205 is a device including a microphone that records the voice of the user from the location O to the location R. 1 A voice presentation device 204 is installed at a site R that reproduces and outputs the voice transmitted to the 1 The return voice recording device 205 is an example of a voice recording device.

[0030] An example of the configuration of the server 2 will be described. The server 2 includes a control unit 21, a program storage unit 22, a data storage unit 23, a communication interface 24, and an input / output interface 25. The elements included in the server 2 are connected to each other via a bus. The control unit 21 may be configured in the same manner as the control unit 11. The processor loads a program stored in the ROM or the program storage unit 22 into the RAM. The processor executes the program loaded into the RAM, causing the control unit 21 to realize each of the functional units described below. The control unit 21 constitutes a computer. Program storage unit 22 may be configured similarly to program storage unit 12 . Data store 23 may be configured similarly to data store 13 . The communication interface 24 may be configured similarly to the communication interface 14. The communication interface 14 includes various interfaces that communicatively connect the server 2 to other electronic devices. The input / output interface 25 may be configured similarly to the input / output interface 15. The input / output interface 25 enables communication between the server 2 and each of the video presentation device 201, the offset video shooting device 202, the loopback video shooting device 203, the audio presentation device 204, and the loopback audio recording device 205. The hardware configuration of the server 2 is not limited to the above-mentioned configuration. The server 2 allows the above-mentioned components to be omitted or changed, and new components to be added, as appropriate. In addition, base R 2 ~Base R n The hardware configuration of the multiple electronic devices included in each of the above-mentioned sites R 1 Since this is the same as the above, the description thereof will be omitted.

[0031] The time distribution server 10 is an electronic device that manages a reference system clock. The reference system clock is absolute time.

[0032] FIG. 2 is a block diagram showing an example of the software configuration of each electronic device that constitutes the media processing system S according to the first embodiment.

[0033] The server 1 includes a time management unit 111, an event video transmission unit 112, a return video reception unit 113, a video processing notification unit 114, an event audio transmission unit 115, a return audio reception unit 116, and an audio processing notification unit 117. Each functional unit is realized by execution of a program by the control unit 11. It can also be said that each functional unit is included in the control unit 11 or the processor. Each functional unit can be interpreted as the control unit 11 or the processor.

[0034] The time management unit 111 performs time synchronization with the time distribution server 10 using a known protocol such as NTP or PTP, and manages the reference system clock. The time management unit 111 manages a reference system clock that is the same as the reference system clock managed by the server 2. The reference system clock managed by the time management unit 111 and the reference system clock managed by the server 2 are time-synchronized.

[0035] The event video transmission unit 112 transmits the video V output from the event video shooting device 101 via an IP network. signal1 The RTP packet containing 1 ~Base R n The video is sent to each server. signal1 is the absolute time T video This is the image captured on . signal1 Acquiring the video V signal1 This includes filming the footage. signal1 Acquiring the video V captured by the event video capture device 101 signal1 This includes sampling the video V signal1 The RTP packet containing video At time T video is video V at base O signal1 is the time when the image was captured. signal1 is an example of the first image. video is an example of a first time. An RTP packet is an example of a packet.

[0036] The return video receiving unit 113 receives the video V signal2 Image V generated from signal3 The RTP packet containing 1 ~Base R n Each server receives the video. signal2 is a video V signal1 Based in R 1 ~Base R n The video is captured at this location at the time it is played at any one of the locations. signal2 The acquisition of the image V signal2 This includes filming the footage. signal2 The acquisition of the image V captured by the image capture device 203 is signal2 This includes sampling the video V signal2 is an example of the second image. signal3 is Δd x_video Based on the processing conditions, base R 1 ~Base R n Each server of the video V signal2 The image generated from the image V signal3 is an example of the third image. signal3 The RTP packet containing video It has been given. signal3 is video V signal2 Since it is generated from signal3 The RTP packet containing the video V signal2 Here is an example of a packet for Δd x_video is base O and base R 1 ~Base R n Δd is a value related to the data transmission delay between each of the x_video is an example of a transmission delay time. x_video is base R 1 ~Base R n Each is different.

[0037] The video processing notification unit 114 is 1 ~Base R n For each of Δd x_video Generate Δd x_videoThe RTCP packet containing 1 ~Base R n Δd x_video The RTCP packet storing the is an example of a notification regarding a transmission delay time. The RTCP packet is an example of a packet.

[0038] The event audio transmission unit 115 receives the audio A output from the event audio recording device 103 via the IP network. signal1 The RTP packet containing 1 ~Base R n The audio is sent to each server. signal1 is the absolute time T audio Audio A was recorded. signal1 Acquiring the voice A signal1 This includes recording Audio A signal1 Acquiring the voice A recorded by the event voice recording device 103 signal1 This involves sampling the audio A. signal1 The RTP packet containing audio At time T audio is voice A at base O signal1 is the time when the audio was acquired. signal1 is an example of the first sound. audio is an example of the first time.

[0039] The return voice receiving unit 116 receives the voice A via the IP network. signal2 Speech A generated from signal3 The RTP packet containing 1 ~Base R n Received from each server. Audio A signal2 is Voice A signal1 Based in R 1 ~Base R n This is the audio recorded at this location at the time it is played at any of the locations. signal2 The acquisition of the voice A is performed by the return voice recording device 205. signal2 This includes recording Audio Asignal2 The acquisition of the voice A recorded by the return voice recording device 205 is signal2 This involves sampling the audio A. signal2 is an example of the second voice. signal3 is Δd x_audio Based on the processing conditions, base R 1 ~Base R n Each server of Voice A signal2 The speech generated from the following: signal3 is an example of the third voice. signal3 The RTP packet containing audio It is given. Audio A signal3 is Audio A signal2 Since it is generated from signal3 The RTP packet containing the audio A signal2 Here is an example of a packet for Δd x_audio is base O and base R 1 ~Base R n Δd is a value related to the data transmission delay between each of the x_ audio is an example of a transmission delay time. x_ audio is base R 1 ~Base R n Each is different.

[0040] The voice processing and notification unit 117 1 ~Base R n For each of Δd x_ audio Generate Δd x_ audio The RTCP packet containing 1 ~Base R n Δd x_ audio An RTCP packet containing this is an example of a notification regarding transmission delay time.

[0041] The server 2 includes a time management unit 2101, an event video receiving unit 2102, a video offset calculation unit 2103, a video processing receiving unit 2104, a return video processing unit 2105, a return video transmitting unit 2106, an event audio receiving unit 2107, an audio processing receiving unit 2108, a return audio processing unit 2109, a return audio transmitting unit 2110, a video time management DB 231, and an audio time management DB 232. Each functional unit is realized by execution of a program by the control unit 21. It can also be said that each functional unit is provided by the control unit 21 or the processor. Each functional unit can be read as the control unit 21 or the processor. The video time management DB 231 and the audio time management DB 232 are realized by the data storage unit 23.

[0042] The time management unit 2101 performs time synchronization with the time distribution server 10 using a known protocol such as NTP or PTP, and manages the reference system clock. The time management unit 2101 manages a reference system clock that is the same as the reference system clock managed by the server 1. The reference system clock managed by the time management unit 2101 and the reference system clock managed by the server 1 are time-synchronized.

[0043] The event video receiving unit 2102 receives the video V signal1 The event video receiving unit 2102 receives an RTP packet storing the video V from the server 1. signal1 to the image presentation device 201. The event image receiving unit 2102 is an example of a second receiving unit. The image offset calculation unit 2103 calculates the image V signal1 The presentation time t 1 The image offset calculation unit 2103 is an example of a calculation unit. The video processing and receiving unit 2104 x_video The video processing receiving unit 2104 receives an RTCP packet storing the above from the server 1. The video processing receiving unit 2104 is an example of a first receiving unit. The folded image processing unit 2105 converts Δd x_video Depending on the processing state based on the video V signal2 Video from V signal3The folded image processing unit 2105 is an example of a processing unit. The return video transmission unit 2106 returns the video V signal3 The RTP packet containing the video V is sent to Server 1. signal3 The RTP packet containing the video V signal2 The presentation time t coincides with the absolute time t at which the image was taken. 1 The time T associated with video The return video transmission unit 2106 is an example of a transmission unit.

[0044] The event voice receiving unit 2107 receives voice A via an IP network. signal1 The event voice receiving unit 2107 receives an RTP packet storing the voice A from the server 1. signal1 to the audio presentation device 204. The event audio receiving unit 2107 is an example of a second receiving unit. The voice processing and receiving unit 2108 x_audio The voice processing receiving unit 2108 receives an RTCP packet storing the above from the server 1. The voice processing receiving unit 2108 is an example of a first receiving unit. The loopback voice processing unit 2109 performs the following operation. x_audio Depending on the processing state based on signal2 Audio A signal3 The return voice processing unit 2109 is an example of a processing unit. The return voice transmission unit 2110 transmits the voice A via the IP network. signal3 The RTP packet containing the above is sent to Server 1. signal3 The RTP packet containing audio The return voice transmitting unit 2110 is an example of a transmitting unit.

[0045] FIG. 3 is a diagram showing the location R according to the first embodiment. 1 2 is a diagram showing an example of a data structure of a video time management DB 231 included in the server 2. FIG. The video time management DB 231 stores the time T video and presentation time t 1It is a database that stores and associates the above. The video time management DB 231 includes a video synchronization reference time column and a presentation time column. video The presentation time column stores the presentation time t 1 Store.

[0046] FIG. 4 is a diagram showing the location R according to the first embodiment. 1 2 is a diagram showing an example of a data structure of an audio time management DB 232 included in the server 2. FIG. The voice time management DB 232 stores the time T audio and Audio A signal1 It is a database that stores and associates the above. The audio time management DB 232 includes an audio synchronization reference time column and an audio data column. audio The voice data column stores voice A signal1 Store.

[0047] In addition, base R 2 ~Base R n Each server of the base R 1 It includes the same functional parts and DB as Server 1 in Base R 1 The same process is performed as for Server 1 of Site R. 2 ~Base R n The processing flow and DB structure of the functional parts included in each server will not be described.

[0048] (Example of operation) In the following, base O and base R 1 The operation of the following will be explained as an example. 2 ~Base R n The operation of the base R 1 The operation may be the same as that of the base R, and the explanation thereof will be omitted. 1 The notation is base R 2 ~Base R n It may also be read as follows.

[0049] (1) Processing and playback of loopback video The video processing of the server 1 in the location O will be described. FIG. 5 is a flowchart showing the procedure and contents of video processing by the server 1 at the site O according to the first embodiment. The event video transmission unit 112 transmits the video V signal1 The RTP packet containing 1 The server 2 transmits the request to the server 2 (step S11). A typical example of the process of step S11 will be described later. The return video receiving unit 113 receives the video V signal3 The RTP packet containing 1 The received data is received from the server 2 (step S12). A typical example of the process in step S12 will be described later. The video processing notification unit 114 is 1 Regarding Δd x_video Generate Δd x_video The RTCP packet containing 1 The server 2 transmits the request to the server 2 (step S13). A typical example of the process in step S13 will be described later.

[0050] Base R 1 The video processing of the server 2 in the above will be described. FIG. 6 is a diagram showing the location R according to the first embodiment. 1 1 is a flowchart showing the procedure and content of video processing by the server 2 in the video processing apparatus 100. The event video receiving unit 2102 receives the video V signal1 The RTP packet storing the above information is received from the server 1 (step S14). A typical example of the process in step S14 will be described later. The image offset calculation unit 2103 calculates the image V signal1 The presentation time t 1 A typical example of the process in step S15 will be described later. The video processing and receiving unit 2104 x_video The RTCP packet storing the above information is received from the server 1 (step S16). A typical example of the process in step S16 will be described later. The folded image processing unit 2105 converts Δd x_video Depending on the processing state based on the video V signal2 Video from V signal3 A typical example of the process in step S17 will be described later. The return video transmission unit 2106 returns the video V signal3 The RTP packet storing the above information is transmitted to the server 1 (step S18). A typical example of the process of step S18 will be described later.

[0051] The following describes typical examples of the processes of steps S11 to S13 of the above-mentioned server 1 and steps S14 to S18 of the above-mentioned server 2. In order to describe the processes in chronological order, the processes will be described in the order of step S11 of the server 1, step S14 of the server 2, step S15 of the server 2, step S12 of the server 1, step S13 of the server 1, step S16 of the server 2, step S17 of the server 2, and step S18 of the server 2.

[0052] FIG. 7 shows the video V of the server 1 at the site O according to the first embodiment. signal1 7 is a flowchart showing the procedure and processing contents of a transmission process of an RTP packet storing the above-mentioned data. The event video transmission unit 112 receives the video V output from the event video shooting device 101. signal1 At regular intervals I video (step S111). The event video transmission unit 112 transmits the video signal1 In step S112, for example, the event video transmission unit 112 generates an RTP packet storing the acquired video V signal1 The event video transmission unit 112 stores the video V in an RTP packet based on the reference system clock managed by the time management unit 111. signal1 The absolute time T video The event video transmission unit 112 receives the obtained time T videois stored in the header extension area of ​​the RTP packet. The event video transmission unit 112 transmits the generated video V signal1 The RTP packet storing the above is sent to the IP network (step S113).

[0053] FIG. 8 is a diagram showing the location R according to the first embodiment. 1 Server 2 video V signal1 8 is a flowchart showing a procedure and process contents of a reception process of an RTP packet storing the above-mentioned information. The event video receiving unit 2102 receives the video V transmitted from the event video transmitting unit 112 via the IP network. signal1 The RTP packet storing the above is received (step S141). The event video receiving unit 2102 receives the video V signal1 The video V stored in the RTP packet signal1 (step S142). The event video receiving unit 2102 receives the acquired video V signal1 The image display device 201 outputs the image V signal1 Play and display. The event video receiving unit 2102 receives the video V signal1 The time T stored in the header extension area of ​​the RTP packet video is obtained (step S144). The event video receiving unit 2102 receives the acquired video V signal1 and time T video to the video offset calculation unit 2103 (step S145).

[0054] FIG. 9 is a diagram showing the location R according to the first embodiment. 1 Server 2's presentation time t 1 9 is a flowchart showing a calculation process procedure and process contents of .times. ... The image offset calculation unit 2103 calculates the image V signal1 and time T videois acquired from the event video receiving unit 2102 (step S151). The image offset calculation unit 2103 calculates the offset of the acquired image V signal1 Based on the image input from the offset image capture device 202, the presentation time t 1 In step S152, for example, the image offset calculation unit 2103 calculates an image V from the images captured by the offset image capture device 202 using a known image processing technique. signal1 The video offset calculation unit 2103 extracts a video frame including the shooting time given to the extracted video frame from the presentation time t 1 The shooting time is an absolute time. The video offset calculation unit 2103 calculates the obtained time T video is stored in the video synchronization reference time column of the video time management DB 231 (step S153). The video offset calculation unit 2103 calculates the presentation time t 1 is stored in the presentation time column of the video time management DB 231 (step S154).

[0055] FIG. 10 shows the video V of the server 1 at the site O according to the first embodiment. signal3 10 is a flowchart showing a procedure and process contents of receiving an RTP packet storing the above-mentioned information. The return video receiving unit 113 receives the video V transmitted from the return video transmitting unit 2106 via the IP network. signal3 The RTP packet storing the above is received (step S121). The return video receiving unit 113 receives the video V signal3 The time T stored in the header extension area of ​​the RTP packet video (Step S122). The return video receiving unit 113 receives the video V signal3 From the information stored in the header of the RTP packet, the source site R x (x is any of 1, 2, . . . , n) is obtained (step S123).

[0056] The return video receiving unit 113 receives the video V signal3 The video V stored in the RTP packet signal3 is obtained (step S124). The return video receiving unit 113 receives the video V signal3 In step S125, for example, the return video receiving unit 113 outputs the return video at a certain interval I video Video V signal3 The return image display device 102 outputs the return image to the image display device 102. 1 Video V is transmitted back to point O from signal3 Play and display.

[0057] The return video receiving unit 113 receives the current time T n (Step S126). The current time T n The return video receiving unit 113 receives the video V signal3 The current time T n is a video V signal3 It can also be said that the current time T n is a video V signal3 It can also be said that the playback time of the video V signal3 The current time T n is an example of the second time. The return video receiving unit 113 receives the received time T video , current time T n and source location R x to the video processing notifying unit 114 (step S127).

[0058] FIG. 11 shows the Δd x_video 11 is a flowchart showing a procedure and process contents of a transmission process of an RTCP packet storing the above-mentioned information. The video processing notification unit 114 receives the return video from the return video receiving unit 113 at time T video , current time T n and source location R x (Step S131). The video processing notification unit 114 video and the current time T n Based on the current time T n From time T video Time minus (T n - T video ) is calculated (step S132). The video processing notification unit 114 determines whether or not the time n - T video ) is the current Δd x_video It is determined whether or not it matches (step S133). x_video is the current time T n and time T video The difference between the current Δd x_video is the time calculated this time (T n - T video ) value. n - T video ) value. Note that Δd x_video The initial value of is 0. Time (T n - T video ) is the current Δd x_video If it matches (step S133, YES), the process ends. n - T video ) is the current Δd x_video If it does not match (NO in step S133), the process proceeds from step S133 to step S134. n - T video ) is the current Δd x_video The fact that it does not match Δd x_video corresponds to a change in

[0059] The video processing notification unit 114 x_video Δd x_video = T n - T video (step S134). The video processing notification unit 114x_video In step S135, for example, the video processing notifying unit 114 transmits an RTCP packet storing the updated Δd x_video is described using the APP in RTCP. x_video The video processing notification unit 114 generates an RTCP packet that stores Δd x_video The RTCP packet containing the x The data is sent to the location indicated by .

[0060] FIG. 12 is a diagram showing the location R according to the first embodiment. 1 Δd of server 2 in x_video 12 is a flowchart showing a procedure and content of a receiving process of an RTCP packet storing the received RTCP packet. The video processing and receiving unit 2104 x_video The RTCP packet storing the above is received from the server 1 (step S161). The video processing and receiving unit 2104 x_video Δd stored in the RTCP packet x_video (step S162). The video processing and receiving unit 2104 receives the acquired Δd x_video The image is then passed back to the image processing unit 2105 (step S163).

[0061] FIG. 13 is a diagram showing the location R according to the first embodiment. 1 Server 2 video V signal2 13 is a flowchart showing the processing procedure and processing contents of the above. The loopback video processing unit 2105 receives the signal Δd from the video processing receiving unit 2104. x_video is obtained (step S171). The loopback image processing unit 2105 processes the image V output from the loopback image shooting device 203. signal2 At regular intervals I video (Step S172). signal2 The image display device 201 displays the image V signal1Based in R 1 At the time to play at base R 1 This is footage taken at.

[0062] The folded image processing unit 2105 processes the acquired Δd x_video The captured image V is processed according to the processing state based on signal2 Video from V signal3 In step S173, for example, the aliased image processing unit 2105 generates Δd x_video Based on the video V signal2 The folded image processing unit 2105 determines the processing mode of Δd x_video Based on the video V signal2 The aliased image processing unit 2105 changes the processing mode of Δd x_video The processing mode is changed so that the quality of the image is lowered as the image V signal2 Processing and video V signal2 The processing mode may include not performing any processing on the image V. signal2 The folded image processing unit 2105 processes the image V signal2 When processing the image V signal3 is video V signal2 The loopback image processing unit 2105 processes the image V signal2 If no processing is done on the image V signal3 is video V signal2 is the same as:

[0063] The folded image processing unit 2105 converts Δd x_video Based on this, the video is processed so that the visibility is reduced when the video is played back on the loopback video display device 102 at the site O. The processing to reduce the visibility includes a processing to reduce the data size of the video. signal2 By replaying the image on the image display device 102, the viewer does not feel uncomfortable. x_video If V is small, the aliased image processing unit 2105 signal2 No processing is performed on Δd x_videoEven if V is too large, the aliased image processing unit 2105 adjusts the image V so that the image does not become completely unrecognizable. signal2 For example, the image V signal2 The processing to change the display size of the image V will be described. signal2 If the horizontal pixels are w and the vertical pixels are h, the image V generated according to the processing mode is signal3 The horizontal pixel w' and vertical pixel h' are as follows: (1) 0ms ≦ Δd x_video When ≦ 300 ms w' = w, h' = h (2) 300ms < Δd x_video When ≦ 500 ms w' = {-(1 / 400) * Δd x_video + 7 / 4}*w, h' = {-(1 / 400) * Δd x_video + 7 / 4} * h (3) 500ms <Δd x_video When w' = 0.5 * w, h' = 0.5 * h The processing is not limited to the above as a change in image quality, and may be, in addition to the above-mentioned change in display size, blurring the image with a Gaussian filter, reducing the brightness of the image, etc. signal3 Video V signal2 Other processing may be used as long as it reduces visibility more than the above. The loopback image processing unit 2105 processes the acquired image V signal2 and the generated image V signal3 The received signal is then passed back to the video transmitting unit 2106 (step S174).

[0064] FIG. 14 is a diagram showing the location R according to the first embodiment. 1 Server 2 video V signal3 14 is a flowchart showing a typical example of the process of step S18 of the server 2. The process of step S18 of the server 2 is shown in FIG. The loopback video transmission unit 2106 receives the video V from the loopback video processing unit 2105.signal2 and video V signal3 In step S181, for example, the return video transmission unit 2106 acquires the video V signal2 and video V signal3 At regular intervals I video and obtain them simultaneously.

[0065] The return video transmission unit 2106 transmits the acquired video V signal2 In step S182, for example, the return video transmission unit 2106 calculates a time t that is an absolute time when the video V signal2 The time code T indicates the time of shooting. c If an absolute time is given, t = T c The time t is obtained as the image V signal2 Time code T c If no time is added, the return video transmission unit 2106 calculates the current time T n The return video transmission unit 2106 obtains a predetermined value t video_offset (positive number) t = T n -t video_offset to obtain the time t.

[0066] The return video transmission unit 2106 refers to the video time management DB 231 and detects the time t that matches the acquired time t. 1 (step S183). The return video transmission unit 2106 refers to the video time management DB 231 and extracts the time T video is obtained (step S184). The return video transmission unit 2106 transmits the video V signal3 In step S185, for example, the return video transmission unit 2106 generates an RTP packet storing the acquired video V signal3 The return video transmission unit 2106 stores the obtained time T video is stored in the header extension area of ​​the RTP packet. The return video transmission unit 2106 transmits the generated video V signal3 The RTP packet storing the above is sent to the IP network (step S186).

[0067] (2) Processing and playback of loopback audio The voice processing of the server 1 at the site O will be described. FIG. 15 is a flowchart showing the procedure and contents of voice processing by the server 1 at the site O according to the first embodiment. The event voice transmission unit 115 transmits the voice A via the IP network. signal1 The RTP packet containing 1 The server 2 transmits the request to the server 2 (step S19). A typical example of the process of step S19 will be described later. The return voice receiving unit 116 receives the voice A via the IP network. signal3 The RTP packet containing 1 The received data is received from the server 2 (step S20). A typical example of the process of step S20 will be described later. The voice processing and notification unit 117 1 Regarding Δd x_audio Generate Δd x_ audio The RTCP packet containing 1 The server 2 transmits the request to the server 2 (step S21). A typical example of the process in step S21 will be described later.

[0068] Base R 1 The audio processing of the server 2 in the above will be described. FIG. 16 is a diagram showing the location R according to the first embodiment. 1 13 is a flowchart showing the procedure and content of voice processing by the server 2 in the embodiment. The event voice receiving unit 2107 receives voice A via an IP network. signal1 The RTP packet storing the above information is received from the server 1 (step S22). A typical example of the process of step S22 will be described later. The voice processing and receiving unit 2108 x_audioThe RTCP packet storing the above information is received from the server 1 (step S23). A typical example of the process in step S23 will be described later. The loopback voice processing unit 2109 performs the following operation. x_audio Depending on the processing state based on signal2 Audio A signal3 A typical example of the process in step S24 will be described later. The return voice transmission unit 2110 transmits the voice A via the IP network. signal3 The RTP packet storing the above information is transmitted to the server 1 (step S25). A typical example of the process of step S25 will be described later.

[0069] The following describes typical examples of the processes of steps S19 to S21 of the above-mentioned server 1 and steps S22 to S25 of the above-mentioned server 2. In order to describe the processes in chronological order, the processes will be described in the order of step S19 of the server 1, step S22 of the server 2, step S20 of the server 1, step S21 of the server 1, step S23 of the server 2, step S24 of the server 1, and step S25 of the server 1.

[0070] FIG. 17 shows the voice A of the server 1 at the site O according to the first embodiment. signal1 17 is a flowchart showing the procedure and processing contents of a transmission process of an RTP packet storing the above-mentioned information.

[0071] The event voice transmission unit 115 receives the voice A output from the event voice recording device 103. signal1 At regular intervals I audio (step S191). The event voice transmission unit 115 transmits voice A signal1 In step S192, for example, the event voice transmission unit 115 generates an RTP packet storing the acquired voice A. signal1The event voice transmission unit 115 stores the voice A in the RTP packet based on the reference system clock managed by the time management unit 111. signal1 The absolute time T audio The event voice transmission unit 115 acquires the acquired time T audio is stored in the header extension area of ​​the RTP packet. The event voice transmission unit 115 generates the voice A signal1 The RTP packet storing the above is sent to the IP network (step S193).

[0072] FIG. 18 is a diagram showing the location R according to the first embodiment. 1 Server 2's voice A signal1 18 is a flowchart showing a procedure and process contents of a receiving process of an RTP packet storing the above-mentioned information. The event voice receiving unit 2107 receives the voice A sent from the event voice transmitting unit 115 via the IP network. signal1 The RTP packet storing the above is received (step S221). The event voice receiving unit 2107 receives the voice A signal1 Audio A stored in the RTP packet signal1 (step S222). The event voice receiving unit 2107 receives the acquired voice A signal1 The voice presentation device 204 outputs the voice A to the voice presentation device 204 (step S223). signal1 Play and output. The event voice receiving unit 2107 receives the voice A signal1 The time T stored in the header extension area of ​​the RTP packet audio (step S224). The event voice receiving unit 2107 receives the acquired voice A signal1 and time T audio In step S225, for example, the event voice receiving unit 2107 stores the acquired time T audioThe event voice receiving unit 2107 stores the acquired voice A in the voice synchronization reference time column of the voice time management DB 232. signal1 is stored in the voice data column of the voice time management DB 232.

[0073] FIG. 19 shows a voice A of the server 1 at the site O according to the first embodiment. signal3 19 is a flowchart showing a procedure and process contents of a receiving process of an RTP packet storing the above-mentioned information. The return voice receiving unit 116 receives the voice A sent from the return voice transmitting unit 2110 via the IP network. signal3 The RTP packet storing the above is received (step S201). The return voice receiving unit 116 receives the voice A signal3 The time T stored in the header extension area of ​​the RTP packet audio (Step S202). The return voice receiving unit 116 receives the voice A signal3 From the information stored in the header of the RTP packet, the source site R x (x is 1, 2, . . . , n) is obtained (step S203).

[0074] The return voice receiving unit 116 receives the voice A signal3 Audio A stored in the RTP packet signal3 (Step S204). The return voice receiving unit 116 receives the voice A signal3 to the return voice presentation device 104 (step S205). In step S205, for example, the return voice reception unit 116 outputs the return voice at a certain interval I audio Audio A signal3 The callback speech presentation device 104 outputs the callback speech to the callback speech presentation device 104. 1 Voice A is transmitted back to point O from signal3 Play and display.

[0075] The return voice receiving unit 116 receives the current time T n (Step S206). The current time T n The return voice receiving unit 116 receives the voice A signal3 The current time T n is Voice A signal3 It can also be said that the current time T n is Voice A signal3 This can also be called the playback time of Audio A. signal3 The current time T n is an example of the second time. The return voice receiving unit 116 receives the received time T audio , current time T n and source location R x to the voice processing notifying unit 117 (step S207).

[0076] FIG. 20 shows the Δd x_audio 20 is a flowchart showing a procedure and process contents of a transmission process of an RTCP packet storing the above-mentioned information. The voice processing notifying unit 117 receives the return voice from the return voice receiving unit 116 at time T audio , current time T n and source location R x (Step S211). The audio processing notification unit 117 audio and the current time T n Based on the current time T n From time T audio Time minus (T n - T audio ) is calculated (step S212). The voice processing notification unit 117 determines whether n - T audio ) is the current Δd x_audio It is determined whether or not it matches (step S213). x_audio is the current time T nand time T audio The difference between the current Δd x_ audio is the time calculated this time (T n - T audio ) value. n - T audio ) value. Note that Δd x_audio The initial value of is 0. Time (T n - T audio ) is the current Δd x_audio If it matches (step S213, YES), the process ends. n - T audio ) is the current Δd x_audio If it does not match (step S213, NO), the process proceeds from step S213 to step S214. n - T audio ) is the current Δd x_audio The fact that it does not match Δd x_audio corresponds to a change in

[0077] The voice processing notification unit 117 x_audio Δd x_audio = T n - T audio (step S214). The voice processing notification unit 117 x_audio In step S215, for example, the voice processing notifying unit 117 transmits an RTCP packet storing the updated Δd x_audio is described using the APP in RTCP. x_audio The voice processing notification unit 117 generates an RTCP packet storing Δd x_ audio The RTCP packet containing the x The data is sent to the location indicated by .

[0078] FIG. 21 is a diagram showing the location R according to the first embodiment. 1 Δd of server 2 in x_audio 21 is a flowchart showing a procedure and process contents of a reception process of an RTCP packet storing the received RTCP packet. The voice processing and receiving unit 2108 x_audio The RTCP packet storing the above is received from the server 1 (step S231). The voice processing and receiving unit 2108 x_audio Δd stored in the RTCP packet x_audio (step S232). The voice processing and receiving unit 2108 receives the acquired Δd x_audio The voice is returned to the voice processing unit 2109 (step S233).

[0079] FIG. 22 is a diagram showing the location R according to the first embodiment. 1 Server 2's voice A signal2 22 is a flowchart showing the processing procedure and processing contents of the above. The return voice processing unit 2109 receives the Δd x_audio (step S241). The return voice processing unit 2109 processes the voice A output from the return voice recording device 205. signal2 At regular intervals I audio (Step S242). signal2 The voice presentation device 204 outputs voice A signal1 Based in R 1 At the time to play at base R 1 This is audio captured using the

[0080] The return voice processing unit 2109 uses the acquired Δd x_audio According to the processing state based on signal2 Audio A signal3 In step S243, for example, the folded voice processing unit 2109 generates Δd x_audio Based on Voice A signal2 The loopback voice processing unit 2109 determines the processing mode of Δd x_audio Based on Voice A signal2 The loopback voice processing unit 2109 changes the processing mode of Δd x_audioThe processing mode is changed so that the quality of the voice decreases as the volume increases. signal2 Processing and audio A signal2 The processing mode may include both processing and not processing the voice A. signal2 The return voice processing unit 2109 processes the voice A signal2 When processing audio A signal3 is Audio A signal2 The return voice processing unit 2109 processes the voice A signal2 If no processing is applied to audio A, signal3 is Audio A signal2 is the same as:

[0081] The loopback voice processing unit 2109 performs the following operation. x_audio Based on this, the audio is processed so that the audibility is reduced when the audio is played back by the return audio presentation device 104 at the location O. The processing to reduce the audibility includes processing to reduce the data size of the audio. signal2 The audio presentation device 104 plays back the Δd x_audio If is smaller, the loopback voice processing unit 2109 loops back the voice A signal2 No processing is performed on Δd x_audio Even if the voice A is too loud, the loopback voice processing unit 2109 processes the voice A so that the voice does not become completely inaudible. signal2 For example, audio A signal2 The processing to change the strength of the voice A will be described. signal2 If the strength of is s, the voice A generated according to the processing mode is signal3 The strength s' of is: (1) 0ms ≦ Δd x_audio When ≦ 100ms, s' = s (2) 100ms < Δd x_audio When ≦ 300ms, s' = {- (1 / 400) * Δd x_audio + 5 / 4} * s (3) 300ms < Δd x_audio When s' = 0.5 * s The processing is not limited to the above as a change in the quality of the voice, and may include, in addition to the above sound intensity change, Δd x_audio The high frequency components may be reduced by low-pass filtering such that the threshold value decreases as Δd increases. x_audio The higher the level, the further away the sound seems to come. signal3 is Audio A signal2 Other processing may be used as long as the processing reduces the audibility more than the above. The return voice processing unit 2109 receives the acquired voice A signal2 and generated voice A signal3 The call is returned to the voice transmitting unit 2110 (step S244).

[0082] FIG. 23 is a diagram showing the location R according to the first embodiment. 1 Server 2's voice A signal3 23 is a flowchart showing a typical example of the process of step S25 of the server 2. The process of step S25 of the server 2 is shown in FIG. The return voice transmission unit 2110 receives the voice A from the return voice processing unit 2109. signal2 and Audio A signal3 In step S251, for example, the return voice transmission unit 2110 acquires the voice A signal2 and Audio A signal3 At regular intervals I audio and obtain them simultaneously.

[0083] The return voice transmission unit 2110 refers to the voice time management DB 232 and signal2 The voice A acquired by the return voice transmission unit 2110 is extracted (step S252). signal2 is the audio A reproduced by the audio presentation device 204. signal1 and base R 1 The voice generated at base R 1In step S252, for example, the return voice transmission unit 2110 separates the two voices using a known voice analysis technique. The return voice transmission unit 2110 separates the two voices into voice A and voice B reproduced by the voice presentation device 204 by the separation of the voices. signal1 The return voice transmission unit 2110 refers to the voice time management DB 232 and identifies the voice A reproduced by the identified voice presentation device 204. signal1 The return voice transmission unit 2110 searches for voice data that matches the specified voice A reproduced by the specified voice presentation device 204 by referring to the voice time management DB 232. signal1 Extract records that have audio data that matches.

[0084] The return voice transmission unit 2110 refers to the voice time management DB 232 and extracts the time T audio (step S253). The return voice transmission unit 2110 receives the voice A signal3 In step S254, for example, the return voice transmission unit 2110 generates an RTP packet storing the acquired voice A. signal3 The return voice transmission unit 2110 stores the obtained time T audio is stored in the header extension area of ​​the RTP packet. The return voice transmission unit 2110 transmits the generated voice A signal3 The RTP packet storing the above is sent to the IP network (step S255).

[0085] (effect) As described above, in the first embodiment, the server 2 determines whether the Δd x_video Based on the processing method, the image V signal2 Video from V signal3 Server 2 generates the video V signal3 In a typical example, server 2 sends Δd x_video The server 2 changes the processing mode based on Δd x_videoThe processing mode may be changed so as to lower the quality of the image as the distance increases. In this way, the server 2 can process the image so that the image becomes less noticeable when played back. In general, when viewing an image projected onto a screen or the like from a certain point X, the image can be clearly viewed if the distance from point X to the screen is within a certain range. On the other hand, as the distance increases, the image becomes smaller and blurrier, making it difficult to view.

[0086] Server 2 uses Δd x_audio Depending on the processing state based on the voice A signal2 Audio A signal3 Server 2 generates voice A signal3 In a typical example, server 2 sends Δd x_audio The server 2 changes the processing mode based on Δd x_audio The processing mode may be changed so as to lower the quality of the sound as the frequency increases. In this way, the server 2 can process the sound so that the sound becomes difficult to hear when it is played back. In general, when listening to sound played back from a speaker or the like from a certain point X, if the distance from point X to the speaker (sound source) is within a certain range, the sound can be clearly heard at the same time as the sound source is generated. On the other hand, as the distance increases, the sound travels with a delay from the time of sound playback and is attenuated, making it difficult to hear.

[0087] Server 2 is Δd x_video or Δd x_audio By performing processing to reproduce the above-mentioned viewing experience based on the above, it is possible to convey the state of a viewer at a physically distant location while reducing the sense of discomfort caused by large data transmission delay times.

[0088] In this way, the server 2 can reduce the sense of discomfort felt by the viewer when a plurality of video and audio streams transmitted at different times from a plurality of locations are played back at the location O.

[0089] Furthermore, the server 2 can reduce the data size of the video and audio by processing the video and audio to be transmitted to the location O. This shortens the data transmission time of the video and audio, and reduces the network bandwidth required for data transmission.

[0090] [Second embodiment] In the second embodiment, at a certain remote location R, video and audio transmitted from a location O and video and audio transmitted from a plurality of remote locations other than the location R are played back.

[0091] The time information used to process the video and audio is 1 ~Base R n The time information is stored in the header extension area of ​​the RTP packets transmitted to and from each of the above. For example, the time information is in absolute time format (hh:mm:ss.fff format).

[0092] In the following, two remote locations are 1 and base R 2 The explanation will be centered on the base R 2 In this case, the video and audio transmitted from point O and point R are 1 The process of playing back the video and audio transmitted from the site R at the site O will be described. 1 and base R 2 Receiving and processing the video and audio transmitted back from site R 1 Base R in 2 Receiving and processing video and audio transmitted from base R 2 Base R in 2 The video and audio recorded at sites O and R 1 The description of the transmission process to the .

[0093] Although the video and audio are described as being converted into RTP packets and transmitted, this is not a limitation. Video and audio may be processed and managed by the same functional unit / DB (database). Video and audio may both be stored in a single RTP packet and transmitted.

[0094] (Configuration example) In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted. In the second embodiment, the differences from the first embodiment will be mainly described.

[0095] FIG. 24 is a block diagram showing an example of the hardware configuration of each electronic device included in the media processing system S according to the second embodiment. The media processing system S is a system for processing multiple electronic devices at site O and multiple electronic devices at site R. 1 ~Base R n Each of the bases includes a plurality of electronic devices and a time distribution server 10. The electronic devices and the time distribution server 10 at each base can communicate with each other via an IP network. Similar to the first embodiment, the location O includes the server 1, the event video shooting device 101, and the event audio recording device 103. The location O is an example of a first location.

[0096] Base R 1 Similar to the first embodiment, the site R includes a server 2, an image presentation device 201, an offset image shooting device 202, and an audio presentation device 204. 1 Unlike the first embodiment, the site R is provided with a video shooting device 206 and a voice recording device 207. 1 is an example of a second location. Server 2 is an example of a media processing device. The video camera 206 is located at the base R. 1 For example, the video camera 206 is a device including a camera that captures video from the site O to the site R. 1 A video presentation device 201 is installed at a site R that plays and displays the video transmitted to the 1 The video camera 206 is an example of a video camera. The voice recording device 207 is 1 For example, the voice recording device 207 is a device including a microphone for recording the voice of the user from the site O to the site R. 1 A voice presentation device 204 is installed at a site R that reproduces and outputs the voice transmitted to the 1The voice recording device 207 is an example of a voice recording device.

[0097] Base R 2 The base R includes a server 3, a video presentation device 301, an offset video shooting device 302, an audio presentation device 303, and an offset audio recording device 304. 2 is an example of a third base that is different from the first base and the second base. Server 3 is the base R 2 It is an electronic device that controls each electronic device included in the The image display device 301 is a display device for displaying images from a location O to a location R. 2 Images transmitted to the site R 1 and base R 3 ~Base R n From each of the bases R 2 The video presentation device 301 is an example of a presentation device. The offset video camera 302 is a device capable of recording the shooting time. The offset video camera 302 is a device including a camera installed so as to be able to shoot the entire video display area of ​​the video presentation device 301. The offset video camera 302 is an example of a video camera. The voice presentation device 303 is a device for 2 Voice and location R transmitted to 1 and base R 3 ~Base R n From each of the bases R 2 The audio presentation device 303 is an example of a presentation device. The offset audio recording device 304 is a device capable of recording the recording time. The offset audio recording device 304 is a device including a microphone installed so as to record the audio played back by the audio presentation device 303. The offset audio recording device 304 is an example of an audio recording device.

[0098] An example of the configuration of the server 3 will be described. The server 3 includes a control unit 31, a program storage unit 32, a data storage unit 33, a communication interface 34, and an input / output interface 35. The elements included in the server 3 are connected to each other via a bus. The control unit 31 may be configured in the same manner as the control unit 11. The processor loads a program stored in the ROM or the program storage unit 32 into the RAM. The processor executes the program loaded into the RAM, causing the control unit 31 to realize each of the functional units described below. The control unit 31 constitutes a computer. Program storage unit 32 may be configured similarly to program storage unit 12 . Data store 33 may be configured similarly to data store 13 . The communication interface 34 may be configured similarly to the communication interface 14. The communication interface 34 includes various interfaces that communicatively connect the server 3 to other electronic devices. The input / output interface 35 may be configured similarly to the input / output interface 15. The input / output interface 35 enables communication between the server 3 and each of the video presentation device 301, the offset video shooting device 302, the audio presentation device 303, and the offset audio recording device 304. The hardware configuration of the server 3 is not limited to the above-mentioned configuration. The server 3 allows the above-mentioned components to be omitted or changed, and new components to be added, as appropriate.

[0099] FIG. 25 is a block diagram showing an example of the software configuration of each electronic device constituting the media processing system S according to the second embodiment.

[0100] As in the first embodiment, the server 1 includes a time management unit 111, an event video transmission unit 112, and an event audio transmission unit 115. Each functional unit is realized by execution of a program by the control unit 11. It can also be said that each functional unit is included in the control unit 11 or the processor. Each functional unit can be interpreted as the control unit 11 or the processor.

[0101] The server 2 includes a time management unit 2101, an event video receiving unit 2102, a video offset calculation unit 2103, an event audio receiving unit 2107, a video time management DB 231, and an audio time management DB 232, similar to the first embodiment. The server 2 includes a video processing receiving unit 2111, a video processing unit 2112, a video transmitting unit 2113, an audio processing receiving unit 2114, an audio processing unit 2115, and an audio transmitting unit 2116, unlike the first embodiment. Each functional unit is realized by execution of a program by the control unit 21. It can also be said that each functional unit is included in the control unit 21 or the processor. Each functional unit can be read as the control unit 21 or the processor. The video time management DB 231 and the audio time management DB 232 are realized by the data storage unit 23.

[0102] The video processing and receiving unit 2111 receives Δd x_video The RTCP packet containing 2 ~Base R n Δd x_video is base R 1 and base R 2 ~Base R n Δd is a value related to the data transmission delay between each of the x_video is an example of a transmission delay time. x_video is base R 2 ~Base R n Each of them is different. Δd x_video The RTCP packet storing this is an example of a notification regarding a transmission delay time. The RTCP packet is an example of a packet. The video processing receiving unit 2111 is an example of a first receiving unit.

[0103] The image processing unit 2112 x_video Depending on the processing state based on the video V signal2 Video from V signal3 Generate the image V signal2 is a video V signal1 Based in R 1 At the time to play at base R 1 This is the image captured by the camera. signal2 The acquisition of the image V signal2This includes filming the footage. signal2 Acquiring the image V captured by the image capture device 206 signal2 This includes sampling the video V signal2 is an example of the second image. signal3 is an example of the third image. The image processing unit 2112 is an example of a processing unit.

[0104] The video transmission unit 2113 transmits the video V signal3 The RTP packet containing 2 ~Base R n The video is sent to one of the servers. signal3 The RTP packet containing video It has been given. signal3 The RTP packet containing the video V signal3 The presentation time t coincides with the absolute time t at which the image was taken. 1 The time T associated with video Including. Video V signal3 is video V signal2 Since it is generated from signal3 The RTP packet containing the video V signal2 An RTP packet is an example of a packet. A video transmitting unit 2113 is an example of a transmitting unit.

[0105] The voice processing and receiving unit 2114 x_audio The RTCP packet containing 2 ~Base R n Δd x_audio is base R 1 and base R 2 ~Base R n Δd is a value related to the data transmission delay between each of the x_ audio is an example of a transmission delay time. x_ audio is base R 2 ~Base R n Each of them is different. Δd x_ audio The RTCP packet storing this is an example of a notification regarding a transmission delay time. The voice processing receiving unit 2114 is an example of a first receiving unit.

[0106] The audio processing unit 2115 x_audio Depending on the processing state based on signal2 Audio A signal3 Generate Voice A signal2 is Voice A signal1 Based in R 1 At the time to play at base R 1 The audio was recorded using the following audio: signal2 The acquisition of the voice A signal2 This includes recording Audio A signal2 Acquiring the voice A recorded by the voice recording device 207 signal2 This involves sampling the audio A. signal2 is an example of the second voice. signal3 is an example of the third voice. The voice processing unit 2115 is an example of a processing unit.

[0107] The audio transmission unit 2116 transmits the audio A via the IP network. signal3 The RTP packet containing 2 ~Base R n The audio is sent to one of the servers. signal3 The RTP packet containing audio It is given. Audio A signal3 is Audio A signal2 Since it is generated from signal3 The RTP packet containing the audio A signal2 2 is an example of a packet regarding the voice transmission unit 2116. The voice transmission unit 2116 is an example of a transmission unit.

[0108] The server 3 comprises a time management unit 311, an event video receiving unit 312, a video offset calculation unit 313, a video receiving unit 314, a video processing notification unit 315, an event audio receiving unit 316, an audio offset calculation unit 317, an audio receiving unit 318, an audio processing notification unit 319, a video time management DB 331 and an audio time management DB 332. Each functional unit is realized by execution of a program by the control unit 31. It can also be said that each functional unit is provided by the control unit 31 or the processor. Each functional unit can be interpreted as the control unit 31 or the processor. The video time management DB 331 and the audio time management DB 332 are realized by the data storage unit 33.

[0109] The time management unit 311 performs time synchronization with the time distribution server 10 using a known protocol such as NTP or PTP, and manages the reference system clock. The time management unit 311 manages the same reference system clock as the reference system clock managed by the servers 1 and 2. The reference system clock managed by the time management unit 311 and the reference system clock managed by the servers 1 and 2 are time-synchronized.

[0110] The event video receiving unit 312 receives the video V signal1 The RTP packet containing the video V is received from Server 1. signal1 is the absolute time T video This is the image captured on . signal1 Acquiring the video V signal1 This includes filming the footage. signal1 Acquiring the video V captured by the event video capture device 101 signal1 This includes sampling the video V signal1 The RTP packet containing video At time T video is video V at base O signal1 is the time when the image was captured. signal1 is an example of the first image. video is an example of the first time. The video offset calculation unit 3132 The image display device 301 displays the image V signal1 The presentation time t 1 Calculate the presentation time t 1 is an example of the third time. The video receiving unit 314 receives the video V signal3 The RTP packet containing 1 and base R 3 ~Base R n Received from each server The video processing notification unit 315 is 1 and base R 3 ~Base R n For each of Δd x_video Generate Δd x_video The RTCP packet containing 1 and base R 3 ~Base R n The data is sent to each of the servers.

[0111] The event voice receiving unit 316 receives voice A via an IP network. signal1 Receives an RTP packet containing the following from Server 1: Audio A signal1 is the absolute time T audio Audio A was recorded. signal1 Acquiring the voice A signal1 This includes recording Audio A signal1 Acquiring the voice A recorded by the event voice recording device 103 signal1 This involves sampling the audio A. signal1 The RTP packet containing audio At time T audio is voice A at base O signal1 is the time when the audio was acquired. signal1 is an example of the first sound. audio is an example of the first time. The audio offset calculation unit 317 2 The voice presentation device 303 outputs voice A. signal1 The presentation time t2 Calculate the presentation time t 2 is an example of the third time. The voice receiving unit 318 receives voice A via the IP network. signal3 The RTP packet containing 1 and base R 3 ~Base R n The data is received from each of the servers. The voice processing and notification unit 319 is 1 and base R 3 ~Base R n For each of Δd x_ audio Generate Δd x_ audio The RTCP packet containing 1 and base R 3 ~Base R n The data is sent to each of the servers.

[0112] The video time management DB 331 may have the same data structure as the video time management DB 231. video and presentation time t 1 It is a database that stores and associates the above.

[0113] FIG. 26 is a diagram showing the location R according to the second embodiment. 2 13 is a diagram showing an example of a data structure of an audio time management DB 332 included in the server 3. FIG. The audio time management DB 332 stores the time T audio and presentation time t 2 It is a database that stores and associates the above. The audio time management DB 332 includes an audio synchronization reference time column and a presentation time column. audio The presentation time column stores the presentation time t 2 Store.

[0114] (Example of operation) In the following, base O and base R 1 and base R 2 The operation of the above will be described as an example.

[0115] (1) Video processing and playback The video processing of the server 1 in the location O will be described. The event video transmission unit 112 transmits the video V signal1 The RTP packet containing 1 ~Base R n The video is sent to each server. signal1 The RTP packet containing video At time T video is each base other than base O (R 1 , R 2 , … , R n ) is time information used to process the video. The process of the event video transmission unit 112 may be the same as the process described in the first embodiment with reference to FIG. 7, and the description thereof will be omitted.

[0116] Base R 1 The video processing of the server 2 in the above will be described. FIG. 27 shows the location R according to the second embodiment. 1 1 is a flowchart showing the procedure and content of video processing by the server 2 in the video processing apparatus 100. The event video receiving unit 2102 receives the video V signal1 The RTP packet storing the above is received from the server 1 (step S26). A typical example of the process of the event video receiving unit 2102 in step S26 may be similar to the process described in the first embodiment with reference to FIG. 8, and therefore a description thereof will be omitted.

[0117] The image offset calculation unit 2103 calculates the image V signal1 The presentation time t 1 is calculated (step S27). A typical example of the process of the image offset calculation unit 2103 in step S27 may be similar to the process described in the first embodiment with reference to FIG. 9, and therefore a description thereof will be omitted.

[0118] The video processing and receiving unit 2111 receives Δdx_video The RTCP packet storing the above is received from the server 3 (step S28). A typical example of the process of the video processing receiving unit 2111 in step S28 may be similar to the process of the video processing receiving unit 2104 described in the first embodiment with reference to FIG. In the explanation using Figure 12, the terms "video processing receiving unit 2104," "return video processing unit 2105," and "server 1" are replaced with "video processing receiving unit 2111," "video processing unit 2112," and "server 3," and the explanation of the processing of the video processing receiving unit 2111 will be omitted.

[0119] The image processing unit 2112 x_video Depending on the processing state based on the video V signal2 Video from V signal3 is generated (step S29). A typical example of the process of the video processing unit 2112 in step S29 may be similar to the process of the return video processing unit 2105 described in the first embodiment with reference to FIG. In the description using FIG. 13, the terms “video processing receiving unit 2104”, “return video processing unit 2105”, “return video shooting device 203”, “location O”, and “return video presentation device 102” are changed to “video processing receiving unit 2111”, “video processing unit 2112”, “video shooting device 206”, and “location R 2 " and "image presentation device 301," and the description of the processing of the image processing unit 2112 will be omitted.

[0120] The video transmission unit 2113 transmits the video V signal3 The RTP packet storing the above is transmitted to the server 3 (step S30). A typical example of the process of the video transmission unit 2113 in step S30 may be similar to the process of the return video transmission unit 2106 described in the first embodiment with reference to FIG. In the description using Figure 14, the notations "return video processing unit 2105" and "return video transmission unit 2106" are replaced with "video processing unit 2112" and "video transmission unit 2113", and the description of the processing of the video transmission unit 2113 is omitted.

[0121] Base R 2 The video processing of the server 3 in the above will be described. FIG. 28 shows the location R according to the second embodiment. 2 1 is a flowchart showing the procedure and content of video processing by the server 3 in the video processing apparatus 100. The event video receiving unit 312 receives the video V signal1 The RTP packet storing the above is received from the server 1 (step S31). A typical example of the process of the event video receiving unit 312 in step S31 may be similar to the process of the event video receiving unit 2102 described in the first embodiment with reference to FIG. In the description using Figure 8, the notations "event video receiving unit 2102", "video offset calculation unit 2103", and "video presentation device 201" are replaced with "event video receiving unit 312", "video offset calculation unit 313", and "video presentation device 301", and the description of the processing of event video receiving unit 312 is omitted.

[0122] The image offset calculation unit 313 calculates the image V signal1 The presentation time t 1 is calculated (step S32). A typical example of the process of the image offset calculation unit 313 in step S32 may be similar to the process of the image offset calculation unit 2103 described in the first embodiment with reference to FIG. In the description using Figure 9, the terms "event video receiving unit 2102", "video offset calculation unit 2103", "offset video shooting device 202", and "video time management DB 231" are replaced with "event video receiving unit 312", "video offset calculation unit 313", "offset video shooting device 302", and "video time management DB 331", thereby omitting explanation of the processing of video offset calculation unit 313.

[0123] The video receiving unit 314 receives the video V signal3 The RTP packet containing 1 The request is received from the server 2 (step S33). A typical example of the process of the video receiving unit 314 in step S33 may be similar to the process of the return video receiving unit 113 described in the first embodiment with reference to FIG. In the explanation using Figure 10, the terms "time management unit 111," "callback video receiving unit 113," "video processing notification unit 114," "callback video presentation device 102," and "callback video transmitting unit 2106" are replaced with "time management unit 311," "video receiving unit 314," "video processing notification unit 315," "video presentation device 301," and "video transmitting unit 2113," thereby omitting explanation of the processing of video receiving unit 314.

[0124] The video processing notification unit 315 is 1 Regarding Δd x_video Generate Δd x_video The RTCP packet containing 1 The received message is sent to the server 1 (step S34).

[0125] FIG. 29 shows the location R according to the second embodiment. 2 Δd of server 3 in x_video 29 is a flowchart showing a procedure and process contents of a transmission process of an RTCP packet storing the above-mentioned information. The video processing notification unit 315 receives the video from the video receiving unit 314 at time T video , current time T n and source location R x is obtained (step S341). The video processing notification unit 315 refers to the video time management DB 331 and video Then, a record having a video synchronization reference time that matches the video synchronization reference time is extracted (step S342). The video processing notification unit 315 refers to the video time management DB 331 and extracts the presentation time t 1(Step S343). 1 is at location O at time T video Footage captured on V signal1 Based in R 2 This is the time when the image was reproduced on the image presentation device 301.

[0126] The video processing notification unit 315 notifies the current time T n and presentation time t 1 Based on the current time T n From presentation time t 1 Time minus (T n -t 1 ) is calculated (step S344). The video processing notification unit 315 determines whether or not the time n -t 1 ) is the current Δd x_video It is determined whether or not it matches (step S345). x_video is the current time T n and presentation time t 1 The difference between the current Δd x_video is the time calculated this time (T n -t 1 ) before the calculated time (T n -t 1 ) Note that Δd x_video The initial value of is 0. Time (T n -t 1 ) is the current Δd x_video If it matches (step S345, YES), the process ends. n -t 1 ) is the current Δd x_video If it does not match (step S345, NO), the process proceeds from step S345 to step S346. n -t 1 ) is the current Δd x_video The fact that it does not match Δd x_video corresponds to a change in

[0127] The video processing notification unit 315 x_video Δd x_video = T n -t 1(step S346). The video processing notification unit 315 x_video In step S347, for example, the video processing notification unit 315 transmits an RTCP packet storing the updated Δd x_video is described using the APP in RTCP. x_video The video processing notification unit 315 generates an RTCP packet that stores Δd x_video The RTCP packet containing the x Base R indicated by 1 Send to.

[0128] (2) Audio processing and playback The voice processing of the server 1 at the site O will be described. The event voice transmission unit 115 transmits the voice A via the IP network. signal1 The RTP packet containing 1 ~Base R n The audio is sent to each server. signal1 The RTP packet containing audio At time T audio is each base other than base O (R 1 , R 2 , … , R n The process of the event voice transmission unit 115 may be the same as the process described in the first embodiment with reference to FIG. 17, and the description thereof will be omitted.

[0129] Base R 1 The audio processing of the server 2 in the above will be described. FIG. 30 shows the location R according to the second embodiment. 1 13 is a flowchart showing the procedure and content of voice processing by the server 2 in the embodiment. The event voice receiving unit 2107 receives voice A via an IP network. signal1 The RTP packet storing the above is received from the server 1 (step S35). A typical example of the process of the event voice receiving unit 2107 in step S35 may be similar to the process described in the first embodiment with reference to FIG. 18, and therefore a description thereof will be omitted.

[0130] The voice processing and receiving unit 2114 x_audio The RTCP packet storing the above is received from the server 3 (step S36). A typical example of the process of the voice processing receiving unit 2114 in step S36 may be similar to the process of the voice processing receiving unit 2108 described in the first embodiment with reference to FIG. In the description using Figure 21, the terms "voice processing receiving unit 2108," "return voice processing unit 2109," and "server 1" are replaced with "voice processing receiving unit 2114," "voice processing unit 2115," and "server 3," and the description of the processing of the voice processing receiving unit 2114 will be omitted.

[0131] The audio processing unit 2115 x_audio Depending on the processing state based on signal2 Audio A signal3 is generated (step S37). A typical example of the process of the voice processing unit 2115 in step S37 may be similar to the process of the return voice processing unit 2109 described in the first embodiment with reference to FIG. In the description using FIG. 22, the terms "voice processing receiving unit 2108", "callback voice processing unit 2109", "callback voice recording device 205", "location O" and "callback voice presentation device 104" are changed to "voice processing receiving unit 2114", "voice processing unit 2115", "voice presentation device 204", "location R 2 " and "audio presentation device 303," the description of the processing of the audio processing unit 2115 will be omitted.

[0132] The audio transmission unit 2116 transmits the audio A via the IP network. signal3 The RTP packet storing the above is transmitted to the server 3 (step S38). A typical example of the process of the voice transmitting unit 2116 in step S38 may be similar to the process of the return voice transmitting unit 2110 described in the first embodiment with reference to FIG. In the description using Figure 23, the notations "callback voice processing unit 2109" and "callback voice transmission unit 2110" are replaced with "voice processing unit 2115" and "voice transmission unit 2116", and the description of the processing of the voice transmission unit 2116 is omitted.

[0133] Base R 2 The audio processing of the server 3 in the above will be described. FIG. 31 shows the location R according to the second embodiment. 2 13 is a flowchart showing the procedure and content of voice processing by the server 3 in the embodiment. The event voice receiving unit 316 receives voice A via an IP network. signal1 The RTP packet storing the above information is received from the server 1 (step S39). A typical example of the process in step S39 will be described later.

[0134] The audio offset calculation unit 317 calculates the offset of the audio A from the audio presentation device 303. signal1 The presentation time t 2 A typical example of the process in step S40 will be described later.

[0135] The voice receiving unit 318 receives voice A via the IP network. signal3 The RTP packet containing 1 The request is received from the server 2 (step S41). A typical example of the process of the voice receiving unit 318 in step S41 may be similar to the process of the return voice receiving unit 116 described in the first embodiment with reference to FIG. In the description using Figure 19, the terms "callback voice receiving unit 116," "voice processing notification unit 117," "callback voice presentation device 104," and "callback voice transmitting unit 2110" are replaced with "voice receiving unit 318," "voice processing notification unit 319," "voice presentation device 303," and "voice transmitting unit 2116," thereby omitting explanation of the processing of the voice receiving unit 318.

[0136] The voice processing and notification unit 319 is 1 Regarding Δd x_ audio Generate Δd x_audio The RTCP packet containing 1 The server 1 transmits the request to the server 1 (step S42). A typical example of the process of step S42 will be described later.

[0137] FIG. 32 shows the location R according to the second embodiment. 2 Server 3's voice A signal1 32 is a flowchart showing a procedure and process contents of receiving an RTP packet storing the above-mentioned information. The event voice receiving unit 316 receives the voice A sent from the event voice transmitting unit 115 via the IP network. signal1 The RTP packet storing the above is received (step S391). The event voice receiving unit 316 receives the voice A signal1 Audio A stored in the RTP packet signal1 is obtained (step S392). The event voice receiving unit 316 receives the acquired voice A signal1 The voice presentation device 303 outputs the voice A to the voice presentation device 303 (step S393). signal1 Play and output. The event voice receiving unit 316 receives the voice A signal1 The time T stored in the header extension area of ​​the RTP packet audio is obtained (step S394). The event voice receiving unit 316 receives the acquired voice A signal1 and time T audio to the audio offset calculation unit 317 (step S395).

[0138] FIG. 33 is a diagram showing the location R according to the second embodiment. 2 Server 3's presentation time t 233 is a flowchart showing a calculation process procedure and process contents of the above-mentioned calculation process. The audio offset calculation unit 317 calculates the audio offset signal1 and time T audio is acquired from the event voice receiving unit 316 (step S401). The audio offset calculation unit 317 calculates the offset of the acquired audio A. signal1 Based on the voice input from the offset voice recording device 304, the presentation time t 2 (Step S402). The voice recorded by the offset voice recording device 304 is converted into the voice A played back by the voice presentation device 303. signal1 and base R 2 The voice generated at base R 2 In step S402, for example, the audio offset calculation unit 317 separates the two audio sounds using a known audio analysis technique. The audio offset calculation unit 317 separates the audio sounds and outputs audio A and B as audio A and B in the audio presentation device 303. signal1 The presentation time t 2 Get the. The audio offset calculation unit 317 calculates the acquired time T audio is stored in the audio synchronization reference time column of the audio time management DB 332 (step S403). The audio offset calculation unit 317 calculates the presentation time t 2 is stored in the presentation time column of the voice time management DB 332 (step S404).

[0139] FIG. 34 shows the location R according to the second embodiment. 2 Δd of server 3 in x_audio 34 is a flowchart showing a procedure and process contents of a transmission process of an RTCP packet storing the above-mentioned information. The voice processing notifying unit 319 receives the voice from the voice receiving unit 318 at time T audio , current time T n and source location R x (Step S421). The voice processing notification unit 319 refers to the voice time management DB 332 andaudio Then, a record having an audio synchronization reference time that matches the audio synchronization reference time is extracted (step S422). The voice processing and notification unit 319 refers to the voice time management DB 332 and extracts the presentation time t 2 (Step S423). 2 is at location O at time T audio Audio A captured in signal1 Based in R 2 This is the time when the audio was played back by the audio presentation device 303.

[0140] The voice processing and notification unit 319 receives the current time T n and presentation time t 2 Based on the current time T n From presentation time t 2 Time minus (T n -t 2 ) is calculated (step S424). The voice processing and notification unit 319 n -t 2 ) is the current Δd x_audio It is determined whether or not it matches (step S425). x_ audio is the current time T n and presentation time t 2 The difference between the current Δd x_ audio is the time calculated this time (T n -t 2 ) before the calculated time (T n -t 2 ) Note that Δd x_audio The initial value of is 0. Time (T n -t 2 ) is the current Δd x_audio If it matches (step S425, YES), the process ends. n -t 2 ) is the current Δd x_audio If it does not match (step S425, NO), the process proceeds from step S425 to step S426. n -t 2 ) is the current Δd x_audio The fact that it does not match Δd x_audiocorresponds to a change in The voice processing notification unit 319 is x_audio Δd x_audio = T n - T audio (step S426). The voice processing notification unit 319 is x_audio In step S427, for example, the voice processing notifying unit 319 transmits an RTCP packet storing the updated Δd x_audio is described using the APP in RTCP. x_audio The voice processing notification unit 319 generates an RTCP packet that stores Δd x_ audio The RTCP packet containing the x The data is sent to the location indicated by .

[0141] (effect) As described above, in the second embodiment, the server 2 determines whether the Δd x_video Based on the processing method, the image V signal2 Video from V signal3 Server 2 generates the video V signal3 In a typical example, server 2 sends Δd x_video The server 2 changes the processing mode based on Δd x_video The processing mode may be changed so as to lower the quality of the image as the distance increases. In this way, the server 2 can process the image so that the image becomes less noticeable when played back. In general, when viewing an image projected onto a screen or the like from a certain point X, the image can be clearly viewed if the distance from point X to the screen is within a certain range. On the other hand, as the distance increases, the image becomes smaller and blurrier, making it difficult to view.

[0142] Server 2 is informed of Δd x_audio Depending on the processing state based on the voice A signal2 Audio A signal3 Server 2 generates the video V signal3In a typical example, server 2 sends Δd x_video The server 2 changes the processing mode based on Δd x_video The processing mode may be changed so as to lower the quality of the sound as the frequency increases. In this way, the server 2 can process the sound so that the sound becomes difficult to hear when it is played back. In general, when listening to sound played back from a speaker or the like from a certain point X, if the distance from point X to the speaker (sound source) is within a certain range, the sound can be clearly heard at the same time as the sound source is generated. On the other hand, as the distance increases, the sound travels with a delay from the time of sound playback and is attenuated, making it difficult to hear.

[0143] Server 2 is Δd x_video or Δd x_video By performing processing to reproduce the above-mentioned viewing experience based on the above, it is possible to convey the state of a viewer at a physically distant location while reducing the sense of discomfort caused by large data transmission delay times.

[0144] In this way, server 2 is 2 This can reduce the sense of discomfort viewers may feel when multiple video and audio streams transmitted at different times from multiple locations are played back.

[0145] Furthermore, server 2 is 2 By processing the video and audio to be transmitted, the data size of the video and audio can be reduced. This shortens the data transmission time for the video and audio, and reduces the network bandwidth required for data transmission.

[0146] [Other embodiments] The media processing device may be realized as a single device as described in the above example, or may be realized as multiple devices with distributed functions.

[0147] The program may be transferred in a state where it is stored in an electronic device, or in a state where it is not stored in an electronic device. In the latter case, the program may be transferred via a network, or in a state where it is recorded on a recording medium. The recording medium is a non-transitory tangible medium. The recording medium is a computer-readable medium. The recording medium may be in any form, such as a CD-ROM or a memory card, as long as it is capable of storing the program and is readable by a computer.

[0148] Although the embodiment of the present invention has been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, in implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted.

[0149] In short, this invention is not limited to the above-mentioned embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining multiple components disclosed in the above-mentioned embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0150] 1 Server 2 Server 3 Server 10. Time distribution server 11 Control section 12 Program memory section 13 Data storage unit 14 Communication Interface 15 Input / Output Interface 21 Control section 22 Program memory section 23 Data storage unit 24 Communication Interface 25 Input / Output Interface 31 Control Unit 32 Program memory section 33 Data storage unit 34 Communication Interface 35 Input / Output Interface 101 Event video recording equipment 102 Reflected image display device 103 Event audio recording device 104 Return voice presentation device 111 Time Management Department 112 Event video transmission unit 113 Return video receiving unit 114 Video Processing Notification Section 115 Event audio transmission unit 116 Return voice receiving unit 117 Voice Processing Notification Department 201 Image display device 202 Offset video equipment 203 Loopback video camera 204 Audio presentation device 205 Call-back voice recording device 206 Video equipment 207 Audio Recording Device 2101 Time Management Department 2102 Event video receiving unit 2103 Image offset calculation unit 2104 Video processing receiver 2105 Return Video Processing Department 2106 Return video transmission unit 2107 Event audio receiver 2108 Voice processing receiver 2109 Return voice processing section 2110 Return voice transmission unit 2111 Video processing receiver 2112 Video Processing Department 2113 Video transmission unit 2114 Voice processing receiver 2115 Audio Processing Department 2116 Voice transmission unit 231 Video time management DB 232 Audio time management DB 301 Video display device 302 Offset video equipment 303 Audio presentation device 304 Offset Audio Recording Device 311 Time Management Department 312 Event video receiving unit 313 Video offset calculation unit 314 Video Receiving Unit 315 Video Processing Notification Department 316 Event Audio Reception Unit 317 Audio offset calculation unit 318 Voice Receiving Unit 319 Voice Processing Notification Department 331 Video time management DB 332 Audio time management DB O Base R 1 ~R n Location S Media Processing System

Claims

1. a media processing device at a second location different from the first location, a first receiving unit that receives from the electronic device at the first location a notification regarding a transmission delay time based on a second time associated with the electronic device at the first location receiving packets related to the media acquired at the second location at a first time when the media is acquired at the first location and a time when the media is to be played at the second location; a second receiving unit that receives a packet storing the first media acquired at the first location from the electronic device at the first location and outputs the first media to a presentation device; a processing unit that generates a third media from the second media acquired at the second location at a time when the first media is to be played at the second location in accordance with a processing mode based on the transmission delay time; a transmission unit that transmits the third media to an electronic device at the first location; A media processing device comprising:

2. the transmission delay time is a difference between the second time and the first time, The processing unit changes the processing mode based on the difference value.

2. The media processing device according to claim 1.

3. a media processing device at a second location different from the first location, a first receiving unit that receives, from an electronic device at a third location, a notification regarding a transmission delay time based on a second time associated with reception by an electronic device at a third location of a packet related to media acquired at the second location at a time when the media acquired at the first time at the first location is to be played at the second location, and a third time when the media acquired at the first time at the first location is played at the third location; a second receiving unit that receives a packet storing the first media acquired at the first location from the electronic device at the first location and outputs the first media to a presentation device; a processing unit that generates a third media from the second media acquired at the second location at a time when the first media is to be played at the second location in accordance with a processing mode based on the transmission delay time; a transmission unit that transmits the third media to an electronic device at the third location; A media processing device comprising:

4. the transmission delay time is a difference between the second time and the third time, The processing unit changes the processing mode based on the difference value. The media processing device according to claim 3.

5. 5. The media processing device according to claim 2, wherein the processing section changes the processing mode so as to degrade the quality of the media as the difference value increases.

6. A media processing method using a media processing device at a second location different from the first location, comprising: receiving a notification from the electronic device at the first location regarding a transmission delay time based on a second time associated with receipt by the electronic device at the first location of packets related to the media acquired at the second location at a first time when the media was acquired at the first location and a time when the media is to be played at the second location; receiving a packet storing a first media acquired at the first location from an electronic device at the first location; outputting the first media to a presentation device; generating a third media from the second media acquired at the second location at a time when the first media is to be played at the second location in accordance with a processing mode based on the transmission delay time; transmitting the third media to an electronic device at the first location; A media processing method comprising:

7. A media processing method using a media processing device at a second location different from the first location, comprising: receiving a notification from an electronic device at the third location regarding a second time associated with receipt by an electronic device at a third location of a packet related to media acquired at the second location at a time when the media acquired at the first time at the first location is to be played at the second location, and a notification regarding a transmission delay time based on a third time when the media acquired at the first time at the first location is played at the third location; receiving a packet storing a first media acquired at the first location from an electronic device at the first location; outputting the first media to a presentation device; generating a third media from the second media acquired at the second location at a time when the first media is to be played at the second location in accordance with a processing mode based on the transmission delay time; transmitting the third media to an electronic device at the third location; A media processing method comprising:

8. A media processing program for causing a computer to execute processing by each unit of the media processing device according to any one of claims 1 to 5.

Citation Information

Patent Citations

  • Method for calibrating video and voice delay of video conference device during using echo canceler

    JP2010171594A

  • External round trip latency measurement for communication systems

    JP2016521470A

  • Remote action guidance system and processing method therefor

    WO2015060393A1