Video / audio reproduction device and program thereof

Abstract

To provide a video / audio reproduction device capable of correcting a mismatch between video and audio.SOLUTION: A video and audio reproduction device 10 includes a guide object placement unit 11, a video renderer 12, an audio renderer 13, a coordinate detection unit 14 that detects a first coordinate of a guide object included in a captured image, a difference calculation unit 15 that calculates the difference between the first coordinate and second coordinate of the guide object, and a delay correction unit 16 that delays the audio reproduced by the audio renderer 13 on the basis of the difference calculated by the difference calculation unit 15.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a video / audio reproducing device and a program for reproducing video and audio in VR. [Background technology]

[0002] Virtual reality (VR) services that support 6DoF (Degree of Freedom), which has six degrees of freedom, are becoming more widespread. These VR services allow users to play VR content from any viewpoint according to their interactive operations.

[0003] In a VR video and audio playback system, a "lag" between video and audio may occur. Several conventional technologies have been proposed to address this lag. Patent Document 1 proposes a method for correcting delays between multiple audio channels. Patent Document 2 proposes a method for correcting synchronization errors on the receiving side when a server transmits video and audio signals. Patent Documents 3 and 4 propose a method for correcting differences in transmission time by using timestamps added to packets when receiving video and audio streams. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-201669 A [Patent Document 2] JP 2005-229168 A [Patent Document 3] JP 2020-149398 A [Patent Document 4] International Publication No. 2020 / 008716 Summary of the Invention [Problem to be solved by the invention]

[0005] Generally, a VR video and audio playback system has independent renderers for video and audio, and differences in the processing delays of each renderer can cause delays between video and audio. However, the above-mentioned conventional technology is a method that assumes transmission delays, but does not assume differences in the processing delays of each renderer, and cannot correct the delays caused by this.

[0006] An object of the present invention is to provide a video / audio reproducing device and a program therefor that can correct the misalignment between video and audio caused by differences in processing delays of each renderer. [Means for solving the problem]

[0007] In order to solve the above problems, the video and audio playback device of the present invention is a video and audio playback device that receives a viewpoint position within a virtual space in response to a user's operation and plays video and audio in synchronization with the input viewpoint position, and is configured to include a guide object placement unit, a video renderer, an audio renderer, a coordinate detection unit, a difference calculation unit, and a delay correction unit.

[0008] According to this configuration, the guide object placement unit places the guide object for coordinate detection in advance in the virtual space. The video renderer receives a video stream and performs video rendering on the input video stream to play back a video corresponding to the viewpoint position. The audio renderer receives an audio stream and performs audio rendering on the input audio stream to reproduce audio corresponding to the viewpoint position.

[0009] The coordinate detection unit receives a captured image of the display means playing back a video, and detects a first coordinate of a guide object included in the input captured image. The first coordinate of the guide object corresponds to a coordinate on the screen that is actually displayed, and includes a processing delay of the video renderer. The difference calculation unit calculates a difference between the second coordinates of the guide object determined by the audio rendering and the first coordinates of the guide object detected by the coordinate detection unit. The second coordinates of the guide object correspond to the coordinates on the screen where the guide object should be originally displayed, and do not include a processing delay of the video renderer. The delay correction unit delays the audio reproduced by the audio renderer based on the difference calculated by the difference calculation unit.

[0010] Since the processing load of video rendering changes depending on the complexity of the video, the screen that is actually displayed shifts from the screen that should be displayed. Also, the processing delay of audio rendering is negligibly small compared to video rendering. Therefore, the video and audio playback device corrects the shift between video and audio using the coordinates of the guide object determined by audio rendering.

[0011] In addition, in order to solve the above-mentioned problems, the video and audio playback device of the present invention is a video and audio playback device that receives an input of a viewpoint position within a virtual space in response to a user's operation, and plays video and audio in synchronization with the input viewpoint position, and is configured to include a guide object placement unit, a video renderer, an audio renderer, a coordinate detection unit, a difference calculation unit, and a delay correction unit.

[0012] According to this configuration, the guide object placement unit places the guide object for coordinate detection in advance in the virtual space. The video renderer receives a video stream and performs video rendering on the input video stream to play back a video corresponding to the viewpoint position. The audio renderer receives an audio stream and performs audio rendering on the input audio stream to reproduce audio corresponding to the viewpoint position.

[0013] The coordinate detection unit receives a captured image of the display means playing back a video, and detects a first coordinate of a guide object included in the input captured image. The first coordinate of the guide object corresponds to a coordinate on the screen that is actually displayed, and includes a processing delay of the video renderer. The difference calculation unit calculates the difference between the second coordinates of the guide object arranged in advance and the first coordinates of the guide object detected by the coordinate detection unit. The second coordinates of the guide object correspond to the coordinates on the screen where the guide object should be displayed, and do not include the processing delay of the video renderer. The delay correction unit delays the audio reproduced by the audio renderer based on the difference calculated by the difference calculation unit.

[0014] Since the processing load of video rendering changes depending on the complexity of the video, the screen that is actually displayed is shifted from the screen that should be displayed. Therefore, the video and audio reproduction device can correct the shift between the video and audio by delaying the audio based on the difference between the first coordinate and the second coordinate of the guide object.

[0015] The present invention can also be realized by a program for causing a computer to function as the above-mentioned video / audio reproducing device. Effect of the Invention

[0016] According to the present invention, it is possible to correct the misalignment between video and audio caused by differences in processing delays of each renderer. [Brief description of the drawings]

[0017] [Figure 1] FIG. 4 is an explanatory diagram for explaining the arrangement of a guide object in the embodiment. [Diagram 2] FIG. 4 is an explanatory diagram for explaining coordinate transformation in the embodiment. [Diagram 3] FIG. 4 is an explanatory diagram illustrating a processing delay in video rendering in an embodiment. [Figure 4] FIG. 4 is an explanatory diagram for explaining detection of a guide object in the embodiment. [Diagram 5] 1 is a block diagram showing a configuration of a video and audio reproduction device according to a first embodiment. [Figure 6] 4 is a flowchart showing the operation of the video and audio reproduction device according to the first embodiment. [Figure 7] FIG. 11 is a block diagram showing the configuration of a video and audio reproduction device according to a second embodiment. [Figure 8] 10 is a flowchart showing the operation of the video and audio reproduction device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. However, each embodiment described below is intended to embody the technical idea of ​​the present invention, and unless otherwise specified, the present invention is not limited to the following. In addition, the same symbols are used for the same means, and the description may be omitted.

[0019] [Principle of correcting the misalignment between video and audio] As a premise for explaining each embodiment, the principle of correcting the misalignment between video and audio will be explained below.

[0020] As shown in Fig. 1, a virtual object (e.g., a person) A is placed in a virtual space SP reproduced in VR, and a virtual viewpoint V is set in advance. The virtual viewpoint V is the position of a virtual camera that virtually captures the virtual space SP, and corresponds to the viewpoint position within the virtual space SP. In addition, the virtual viewpoint V can be interactively operated by a user operating the operation means 2 (Fig. 5) or by a user wearing an HMD changing the posture of his or her head.

[0021] Also, a guide object G for coordinate detection is placed in advance in the virtual space SP. This guide object G is an object for detecting coordinates (first coordinates) on the screen that is actually displayed. For example, the guide object G is preferably an invisible object using an infrared image or a digital watermark so as not to interfere with the user's viewing (see References 1 and 2).

[0022] Reference 1: Kobayashi et al., Approach to copyright protection using digital watermarks, Information Management, 2017, vol. 60, No. 2, pp. 89-99 Reference 2: “MIT and Meta develop technology to embed QR codes invisible to the naked eye,” [online], February 3, 2022, [Retrieved May 27, 2023], Internet<URL:https: / / www.itmedia.co.jp / news / articles / 2202 / 03 / news042.html>

[0023] Furthermore, the guide object G may be a visible object such as a building, a tree, or a ball, so long as it fits in with the presentation of the virtual space SP. The guide object G is arranged so that it is always included in the image even if the virtual viewpoint V changes in various directions. As shown in FIG. 1, each guide object G can be individually identified by, for example, assigning an identification number such as "111" or "211." The i-th guide object G on the x-axis, the j-th guide object G on the y-axis, and the k-th guide object G on the z-axis are identified by G. i,j,k (where i, j, and k are integers equal to or greater than 1). In the example of FIG. 1, the guide object G 1,1,1 ,G 1,2,1 ,…,G i,j,k are arranged at equal intervals in a grid pattern.

[0024] Object A and guide object G in the virtual space SP each have three-dimensional coordinates (x, y, z) along the x-axis, y-axis, and z-axis, with a certain point in the virtual space SP as the origin. When creating the CG, object A and guide object G in the virtual space SP are defined in a world coordinate system arbitrarily set in the virtual space SP.

[0025] Guide Object G i,j,k The world coordinates of are set in advance in the virtual space SP as shown in the following equation (1).

[0026]

number

[0027] A guide object G is displayed at a predetermined position in the virtual space SP by image rendering. i,j,k is rendered. In video rendering, the position of the virtual viewpoint V changes by the following equation (2) depending on the user's interactive operation.

[0028]

number

[0029] As shown in FIG. 2 and the following formula (3), in the video rendering, the guide object G i,j,k The world coordinate G Image,Gi,j,k camera coordinate G´ Image,Gi,j,k =(x´ Image,Gi,j,k ,y´ Image,Gi,j,k ,z´ Image,Gi,j,k ) Note that the camera coordinates are a three-dimensional coordinate system based on a virtual camera placed at a virtual viewpoint V in the virtual space SP.

[0030]

number

[0031] R is a rotation matrix for transforming the coordinate system based on the virtual viewpoint V. In addition, the second term on the right-hand side (t Gi,j,k ,t Gi,j,k ,t Gi,j,k ) is a translation vector to the virtual viewpoint V. The rotation matrix R and the translation vector are set in advance.

[0032] Guide object G located in the user's line of sighti,j,k Consider the case where the guide object G is captured by a virtual camera in the virtual space SP. i,j,k Camera coordinates G´ Image,Gi,j,k However, by interactive manipulation, the camera coordinates G^ Image,Gi,j,k =(x^ Image,Gi,j,k ,y^ Image,Gi,j,k ,z^ Image,Gi,j,k )

[0033]

number

[0034] As shown in Fig. 3, video and audio are rendered separately, and the processing time for each is different. In addition, while the processing delay of video rendering is large, the processing delay of audio rendering is negligibly small, resulting in a misalignment between video and audio. Therefore, as explained below, the misalignment between video and audio is corrected.

[0035] As shown in FIG. 4, after a user performs an interactive operation, the display means 3 is photographed with a photographing camera (e.g., an infrared camera) 4 capable of photographing the guide object G, thereby detecting the first coordinates of the guide object G on the display means 3.

[0036] After interactive operation, the guide object G i,j,k The first coordinate of is the image coordinate (u GImage,i,j,k ,v GImage,i,j,k ) This guide object G i,j,k The first coordinate includes the processing delay of the image rendering and represents the coordinate on the screen that is actually displayed. Note that the image coordinate is the two-dimensional coordinate of the image captured by the virtual camera, that is, the two-dimensional coordinate of the image viewed by the user.

[0037]

number

[0038] After interactive operation, the guide object G i,j,k The normalized image coordinates of (x^ Image,Gi,j,k ,y^ Image,Gi,j,k ) Note that normalized image coordinates are two-dimensional coordinates based on the virtual camera.

[0039] After interactive operation, the guide object G i,j,k Image coordinates (u GImage,i,j,k ,v GImage,i,j,k ) and normalized image coordinates (x^ Image,Gi,j,k ,y^ Image,Gi,j,k The relationship between the guide object G and the guide object G is expressed by the following formulas (6) and (7). i,j,k The normalized image coordinates (x^ Image,Gi,j,k ,y^ Image,Gi,j,k ) to the image coordinates (u GImage,i,j,k ,v GImage,i,j,k ) can be converted to

[0040]

number

number

[0041] Here, f is the focal length of the virtual camera in the virtual space SP (the distance between the virtual camera and object A), and δ u,Gi,j,k is the pixel size of the virtual camera in the horizontal direction, δ v,Gi,j,k represents the pixel size of the virtual camera in the vertical direction. u,Gi,j,k is the horizontal position of the intersection between the optical axis and the image plane (projection plane) in the image coordinate system, and c v,Gi,j,k represents the vertical position of the intersection between the optical axis and the image plane (projection plane) in the image coordinate system.

[0042] In addition, the normalized image coordinates (x^ Image,Gi,j,k ,y^ Image,Gi,j,k ) is the camera coordinate (x^ Image,Gi,j,k ,y^ Image,Gi,j,k ,z^ Image,Gi,j,k), so if the camera coordinates are known, the normalized image coordinates are also known.

[0043] In audio rendering, a sound image is reproduced by stereo reproduction, multi-channel surround reproduction, or binaural reproduction. In audio rendering, similarly to video rendering, the position of the virtual viewpoint V changes by the above-mentioned formula (2) according to an interactive operation by a user.

[0044] In addition, in the sound field reproduced by audio rendering, the guide object G i,j,k The world coordinate G Audio, Gi,j,k =(x Audio,Gi,j,k ,y Audio,Gi,j,k ,z Audio,Gi,j,k ) is the camera coordinate G' as in the above equation (3). Audio, Gi,j,k =(x´ Audio,Gi,j,k ,y´ Audio,Gi,j,k ,z´ Audio,Gi,j,k ) can be converted into the guide object G i,j,k Camera coordinates G´ Audio, Gi,j,k However, by interactive manipulation, the camera coordinates G^ Audio, Gi,j,k =(x^ Audio,Gi,j,k ,y^ Audio,Gi,j,k ,z^ Audio,Gi,j,k ) to

[0045] As shown in the following formulas (8) and (9), after the interactive operation, the guide object G i,j,k The normalized image coordinates (x^ Audio,Gi,j,k ,y^ Audio,Gi,j,k ) to the image coordinates (u GAudio,i,j,k ,v GAudio,i,j,k ), which is defined as the second coordinate. The coordinates of the guide object G determined by the audio rendering do not include the processing delay of the video rendering, and correspond to the coordinates on the screen that should be displayed.

[0046]

number

number

[0047] Here, since a processing delay occurs in the image rendering, the guide object G i,j,k The first coordinate (u GImage,i,j,k ,v GImage,i,j,k ) includes the processing delay of the video rendering. In contrast, the second coordinate (u GAudio,i,j,k ,v GAudio,i,j,k ) does not include the processing delay of image rendering and corresponds to the coordinates on the screen where it should be displayed.

[0048] Therefore, as shown in the following formula (10), the guide object G i,j,k Image coordinates (u GImage,i,j,k ,v GImage,i,j,k ) and the second coordinate (u GAudio,i,j,k ,v GAudio,i,j,k ) the deviation can be corrected according to the difference.

[0049]

number

[0050] Here, after the interactive operation, the guide object G detected from the captured image i,j,k The first coordinate of is expressed as the image coordinate (u^ GImage,i,j,k ,v^ GImage,i,j,k ) will be expressed as

[0051]

number

[0052] Specifically, the guide object G i,j,k The first coordinate (u^ GImage,i,j,k ,v^ GImage,i,j,k ) and the second coordinate (u GAudio,i,j,k ,v GAudio,i,j,k) is converted into an angular difference at a predetermined viewing distance d. The deviation (angular difference between the first coordinate and the second coordinate) that can be recognized by a user at a certain viewing distance d is defined by the following formula (12). Note that the viewing distance d is the distance between the user's naked eye and the display means 3.

[0053]

number

[0054] It is known that if this angle difference exceeds 11 degrees, it exceeds the user's tolerance limit (Reference 3). Therefore, if the angle difference exceeds a preset threshold (for example, 11 degrees), the deviation can be corrected. It goes without saying that the threshold for the angle difference is not limited to 11 degrees.

[0055] Reference 3: S. Komiyama: “An Application of the 3-1 Sound System to HDTV Home Receiver”, ICCE, FAM-19.5, (June 1990)

[0056] Specifically, in order to correct the deviation, the guide object G i,j,k The first coordinate (u^ GImage,i,j,k ,v^ GImage,i,j,k ) and the second coordinate (u GAudio,i,j,k ,v GAudio,i,j,k ) to calculate the delay time (time difference Δt). Specifically, the angle difference is converted to a time difference (video processing delay) Δt using the following equation (13). Then, the audio is delayed by the time difference Δt.

[0057]

number

[0058] If the time difference Δt changes from moment to moment, temporary interruptions or discontinuities in the audio may occur, which may be heard as noise. In this case, the discontinuous parts can be smoothed out using signal processing, or a delay can be applied to each block of audio (a chunk of time during which audio is being produced).

[0059] In the above example, the delay is calculated from the difference between the video rendering and the audio rendering, but the present invention is not limited to this. i,j,k Since the world coordinates of do not include the processing delay of the video rendering, they can be used to calculate the difference, similar to the second coordinates of the guide object G determined by the audio rendering.

[0060] A guide object G placed in advance in the virtual space SP i,j,k The image coordinates of the guide object G are expressed as the image coordinates (u GImage,i,j,k ,v GImage,i,j,k ) can be expressed as

[0061]

number

[0062] After interactive operation, the guide object G detected from the captured image i,j,k The image coordinates (first coordinates) of are the image coordinates (u^ GImage,i,j,k ,v^ GImage,i,j,k ) can be expressed as

[0063] Therefore, as shown in the following formula (15), the guide object G i,j,k Image coordinates (u^ GImage,i,j,k ,v^ GImage,i,j,k ) and the image coordinates (u GImage,i,j,k ,v GImage,i,j,k ) the deviation can be corrected according to the difference.

[0064]

number

[0065] Guide Object G i,j,k Image coordinates (u^ GImage,i,j,k ,v^ GImage,i,j,k ) and the image coordinates (u GImage,i,j,k ,v GImage,i,j,k ) is converted into an angular difference at a predetermined viewing distance d. The deviation (angular difference between the first coordinate and the second coordinate) that can be recognized by a user at a certain viewing distance d is defined by the following formula (16).

[0066]

number

[0067] Specifically, in order to correct the deviation, the guide object G i,j,k The first coordinate (u^ GImage,i,j,k ,v^ GImage,i,j,k ) and the second coordinate (u GImage,i,j,k ,v GImage,i,j,k ) to calculate the delay time (time difference Δt). Specifically, the angle difference is converted to a time difference (video processing delay) Δt using the following equation (17). Then, the audio is delayed by the time difference Δt.

[0068]

number

[0069] (First embodiment) [Outline of the video and audio playback system] An overview of a video / audio reproduction system 1 according to the first embodiment will be described with reference to FIG. The video / audio reproduction system 1 reproduces video and audio as a VR service. In the first embodiment, the video / audio reproduction system 1 corrects the misalignment of video and audio using a first method. As shown in FIG. 5, the video / audio reproduction system 1 includes an operation means 2, a display means 3, a shooting camera 4, a speaker 5, and a video / audio reproduction device 10.

[0070] The operation means 2 allows the user to operate the position and direction of the virtual viewpoint V. For example, the operation means 2 is a joystick, and the user can interactively operate the position and direction of the virtual viewpoint V. The operation means 2 outputs the position and direction of the virtual viewpoint V to the video / audio reproduction device 10.

[0071] The display means 3 displays the video input from the video / audio reproduction device 10, that is, the video at the virtual viewpoint V operated by the user. For example, the display means 3 is a flat display such as a liquid crystal display or an organic EL display.

[0072] The photographing camera 4 photographs the display means 3 playing back the video, and is a camera capable of photographing the guide object G included in the video. For example, the photographing camera 4 is an infrared camera when the guide object G is represented by an infrared video. The photographing camera 4 outputs the photographed image of the display means 3 to the video / audio reproducing device 10.

[0073] The speaker 5 reproduces the sound input from the video / audio reproduction device 10, that is, the sound (sound field) at the virtual viewpoint V operated by the user. For example, the speaker 5 is a stereo speaker.

[0074] [Configuration of video / audio playback device] Next, the configuration of the video / audio reproducing device 10 will be described. The video / audio reproducing device 10 receives a viewpoint position (virtual viewpoint V) in a virtual space SP according to a user's operation, and synchronously reproduces video and audio according to the input viewpoint position. As shown in Fig. 5, the video / audio reproducing device 10 includes a guide object placement unit 11, a video renderer 12, an audio renderer 13, a coordinate detection unit 14, a difference calculation unit 15, and a delay correction unit 16.

[0075] The guide object placement unit 11 places a guide object G for coordinate detection in advance in the virtual space SP. For example, in the guide object placement unit 11, a user manually places the guide object G in the virtual space SP using a mouse and a keyboard (not shown) (see FIG. 1).

[0076] The video renderer 12 receives a video stream and performs video rendering on the input video stream to play back a video corresponding to the viewpoint position. That is, the video renderer 12 renders (draws) a video of a virtual space SP, in which an object A and a guide object G are arranged, captured by a virtual camera at a virtual viewpoint V input from the operation means 2. The video renderer 12 outputs the rendered video to the display means 3.

[0077] The audio renderer 13 receives an audio stream and performs audio rendering on the input audio stream to reproduce audio corresponding to the viewpoint position. In other words, in audio rendering, a sound field at a virtual viewpoint V input from the operation means 2 is reproduced. Then, the audio renderer 13 delays the playback timing of the audio and outputs it to the speaker 5 according to an instruction from a delay correction unit 16 described later. Furthermore, the audio renderer 13 outputs the second coordinates (image coordinates) of the guide object G, which do not include the processing delay of the video rendering, to the difference calculation unit 15, as shown in the above formulas (8) and (9).

[0078] The coordinate detection unit 14 receives a captured image of the display unit 3 playing back a video, and detects the first coordinates of the guide object G included in the input captured image. As shown in FIG. 4, the coordinate detection unit 14 detects the first coordinates of the guide object G. i,j,k When the infrared image is used, the center position of each infrared region included in the captured image is determined by the guide object G. i,j,k The coordinate detection unit 14 detects the first coordinate of the detected guide object G (see the above formulas (5) to (7)). i,j,k The first coordinate is output to the difference calculation unit 15.

[0079] The difference calculation unit 15 calculates the difference between the second coordinates of the guide object G determined by the voice rendering and the first coordinates of the guide object G detected by the coordinate detection unit 14. That is, the difference calculation unit 15 calculates the difference between the second coordinates of the guide object G determined by the voice rendering and the first coordinates of the guide object G detected by the coordinate detection unit 14 using the above-mentioned formula (10). i,j,k and the second coordinate of the guide object G input from the coordinate detection unit 14. i,j,k The difference between the first coordinate and the second coordinate is calculated. The difference calculation unit 15 outputs the calculated difference to the delay correction unit 16.

[0080] The delay correction unit 16 delays the audio reproduced by the audio renderer 13 based on the difference calculated by the difference calculation unit 15. Specifically, the delay correction unit 16 converts the difference calculated by the difference calculation unit 15 into an angle difference at a predetermined viewing distance d using the above formula (12).

[0081] Then, if the converted angle difference exceeds a preset threshold (e.g., 11 degrees), the delay correction unit 16 converts the difference into a time difference Δt using the above formula (13). Furthermore, the delay correction unit 16 delays the audio played back by the audio renderer 13 based on the time difference Δt. For example, the delay correction unit 16 outputs a command to the audio renderer 13 to delay the audio stream by the time difference Δt.

[0082] If the converted angle difference is equal to or less than the threshold, there is no need for the delay correction unit 16 to delay the audio. In this case, after performing audio rendering on the audio stream, the audio renderer 13 outputs the audio to the speaker 5 without delaying the playback timing of the audio.

[0083] [Operation of video and audio playback device] The operation of the video / audio reproducing device 10 will be described with reference to FIG. It is assumed that the guide object G is placed in advance within the virtual space SP. 6, in step S1, a video stream is input to the video renderer 12. Also, an audio stream is input to the audio renderer 13.

[0084] In step S2, the video renderer 12 performs video rendering on the video stream to play back a video corresponding to the virtual viewpoint V. The audio renderer 13 performs audio rendering on the audio stream to play back audio corresponding to the virtual viewpoint V.

[0085] In step S3, the coordinate detection unit 14 receives an image captured by the photographing camera 4 of the display means 3 playing back a video. In step S4, the coordinate detection unit 14 detects the first coordinate of the guide object G in the video rendering by using the above-mentioned equations (5) to (7).

[0086] In step S5, the difference calculation unit 15 acquires the second coordinates of the guide object G from the sound renderer 13 by using the above-mentioned formulas (8) and (9). In step S6, the difference calculation unit 15 uses the above-mentioned formula (10) to calculate the difference between the image coordinates (second coordinates) of the guide object G in the sound rendering and the image coordinates (first coordinates) of the guide object detected from the captured image.

[0087] In step S7, the delay corrector 16 converts the difference calculated in step S7 into an angle difference at a predetermined viewing distance d using the above-mentioned equation (12).

[0088] In step S8, the delay correction unit 16 determines whether or not the angle difference converted in step S7 exceeds a threshold value (for example, 11 degrees). If the angle difference exceeds the threshold value (YES in step S8), the video and audio reproducing device 10 proceeds to the process of step S9. If the angle difference is equal to or smaller than the threshold value (NO in step S8), the video / audio reproducing device 10 returns to the process of step S1.

[0089] In step S9, the delay correction unit 16 converts the difference calculated in step S7 into a time difference Δt using the above-mentioned equation (13). In step S10, the delay correction unit 16 delays the audio reproduced by the audio renderer 13 based on the time difference Δt.

[0090] [Actions and Effects] As described above, the video / audio reproduction device 10 according to the first embodiment can correct the mismatch between video and audio caused by the difference in processing delay between the video renderer 12 and the audio renderer 13.

[0091] Since the processing load of the video rendering changes according to the complexity of the video, the screen that is actually displayed is shifted from the screen that should be displayed. Therefore, the video and audio reproducing device 10 can correct the shift between the video and the audio by delaying the audio based on the difference between the first coordinate and the second coordinate of the guide object G.

[0092] Second embodiment [Configuration of video / audio playback device] The configuration of a video / audio reproducing device 10B according to the second embodiment will be described with respect to the differences from the first embodiment.

[0093] The second embodiment differs from the first embodiment in that the second coordinates of the guide object G determined in advance by the formula (14) are used instead of the second coordinates of the guide object G determined by the voice rendering. That is, the coordinates of the guide object G determined by the formula (14) do not include the processing delay of the video rendering, and therefore can be used to calculate the difference in the same way as the second coordinates of the guide object G determined by the voice rendering.

[0094] As shown in FIG. 7, the video and audio reproduction device 10B includes a guide object placement unit 11B, a video renderer 12, an audio renderer 13B, a coordinate detection unit 14, a difference calculation unit 15B, and a delay correction unit 16B.

[0095] The guide object placement unit 11B outputs the coordinates of the guide object G determined by the above formula (14) as second coordinates to the difference calculation unit 15B. In other respects, the guide object placement unit 11B is similar to that in the first embodiment, and therefore description thereof will be omitted.

[0096] The sound renderer 13B does not output the second coordinates of the guide object G to the difference calculation unit 15B. In other respects, the sound renderer 13B is similar to that in the first embodiment, and therefore a description thereof will be omitted.

[0097] The difference calculation unit 15B calculates the difference between the second coordinate of the guide object G arranged in advance and the first coordinate of the guide object G detected by the coordinate detection unit 14. That is, the difference calculation unit 15B uses the above-mentioned formula (15) to calculate the difference between the second coordinate of the guide object G input from the guide object arrangement unit 11B and the first coordinate of the guide object G input from the coordinate detection unit 14. In other respects, the difference calculation unit 15B is similar to that of the first embodiment, and therefore a description thereof will be omitted.

[0098] The delay correction unit 16B delays the audio reproduced by the audio renderer 13B based on the difference calculated by the difference calculation unit 15B. Specifically, the delay correction unit 16B converts the difference calculated by the difference calculation unit 15B into a time difference Δt using the above-mentioned equations (16) and (17).

[0099] It is known that if the time difference Δt exceeds 400 ms, the audio is perceived as being significantly ahead of the others (Reference 4). Therefore, if the time difference Δt is equal to or greater than a preset threshold (e.g., 400 ms), the audio can be delayed by the time difference Δt. It goes without saying that the threshold for the time difference Δt is not limited to 400 ms.

[0100] Reference 4: Okubo, Oda, Kazan, Komiyama: "Interactive 3D Audio Playback System", AES Tokyo Convention 2001 Proceedings, I-1, pp.238-241 (2001)

[0101] That is, when the time difference Δt exceeds a threshold value (e.g., 400 ms), the delay correction unit 16B delays the audio played by the audio renderer 13B based on the time difference Δt. For example, the delay correction unit 16B outputs a command to the audio renderer 13B to delay the audio stream by the time difference Δt.

[0102] If the converted time difference is equal to or less than the threshold, the delay correction unit 16B does not need to delay the audio. In this case, after performing audio rendering on the audio stream, the audio renderer 13B outputs the audio to the speaker 5 without delaying the playback timing of the audio.

[0103] [Operation of video and audio playback device] The operation of the video / audio reproducing device 10B will be described with reference to FIG. The processes in steps S1 to S4 are similar to those in the first embodiment, and therefore the description thereof will be omitted.

[0104] As shown in FIG. 8, in step S20, the difference calculation unit 15B calculates the difference between the second coordinates of the guide object G arranged in advance and the first coordinates of the guide object G detected by the coordinate detection unit 14, using the above-mentioned formula (15). In step S21, the delay correction unit 16B converts the difference calculated by the difference calculation unit 15B into a time difference Δt using the above equations (16) and (17).

[0105] In step S22, the delay corrector 16B determines whether the time difference Δt converted in step S21 exceeds a threshold value (for example, 400 ms). If the time difference Δt exceeds the threshold value (YES in step S22), the video and audio reproducing device 10B proceeds to the process of step S23. If the time difference Δt is equal to or smaller than the threshold value (NO in step S22), the video and audio reproducing device 10B returns to the process in step S1.

[0106] In step S23, the delay correction unit 16B delays the audio reproduced by the audio renderer 13B based on the time difference Δt.

[0107] [Actions and Effects] As described above, the video / audio reproduction device 10B according to the second embodiment can correct the misalignment between video and audio caused by the difference in processing delay between the video renderer 12 and the audio renderer 13B.

[0108] Since the processing load of the video rendering changes according to the complexity of the video, the screen that is actually displayed is shifted from the screen that should be displayed. Therefore, the video and audio reproducing device 10B can correct the shift between the video and the audio by delaying the audio based on the difference between the first coordinate and the second coordinate of the guide object G.

[0109] (Modification) Although each embodiment has been described in detail above, the present invention is not limited to the above-described embodiments, and includes design modifications and the like within the scope of the present invention.

[0110] In the above-mentioned embodiments, the display means is described as a flat display, but is not limited thereto. For example, the display means may be a device (e.g., HMD) that can project the image of the virtual viewpoint as it is. In this case, the video / audio reproducing device does not need to perform the conversion of the above formulas (8) and (9). Also, a small camera for detecting the guide object may be placed inside the HMD, and the captured image may be output to the video / audio reproducing device.

[0111] In the above-described embodiments, the difference between the first coordinate and the second coordinate is calculated in a two-dimensional coordinate system, but the present invention is not limited to this. In other words, the video / audio reproducing device may calculate the difference between the first coordinate and the second coordinate in a three-dimensional coordinate system.

[0112] In the above-mentioned embodiments, two operations of the video / audio reproducing device have been described, but they may be treated as one operation. In other words, the time difference Δt may be calculated for each of the operations in Fig. 6 and Fig. 8, and the audio may be delayed using the larger time difference Δt.

[0113] In the above-mentioned embodiments, the video / audio reproducing device is described as an independent piece of hardware, but the present invention is not limited to this. For example, the present invention can be realized by a program for making hardware resources such as a CPU, memory, and hard disk of a computer function as the video / audio reproducing device. This program may be distributed via a communication line, or may be written on a recording medium such as a CD-ROM or a flash memory and distributed. [Explanation of symbols]

[0114] 1. Audio-video playback system 2 Operating means 3 Display means 4. Shooting camera 5 Speakers 10, 10B Video and audio playback device 11, 11B Guide object placement section 12 Video Renderer 13,13B Audio Renderer 14 Coordinate detection unit 15,15B Difference calculation part 16,16B Delay compensation section

Claims

1. A video / audio reproducing device that receives a viewpoint position in a virtual space according to a user's operation, and synchronously reproduces video and audio according to the input viewpoint position, a guide object arrangement unit that arranges a guide object for coordinate detection in advance in a virtual space; a video renderer that receives a video stream and performs video rendering on the input video stream to reproduce a video corresponding to the viewpoint position; an audio renderer that receives an audio stream and performs audio rendering on the input audio stream to reproduce audio corresponding to the viewpoint position; a coordinate detection unit to which a captured image of a display unit reproducing the video is input and which detects a first coordinate of a guide object included in the input captured image; A difference calculation unit that calculates a difference between a second coordinate of the guide object determined by the audio rendering and a first coordinate of the guide object detected by the coordinate detection unit; a delay correction unit that delays the audio reproduced by the audio renderer based on the difference calculated by the difference calculation unit; A video / audio reproducing device comprising:

2. The video / audio reproduction device of claim 1, characterized in that the delay correction unit converts the difference calculated by the difference calculation unit into an angular difference at a predetermined viewing distance, and if the converted angular difference exceeds a predetermined threshold, converts the difference into a time difference and delays the audio reproduced by the audio renderer based on the time difference.

3. A video / audio reproducing device that receives a viewpoint position in a virtual space according to a user's operation, and synchronously reproduces video and audio according to the input viewpoint position, a guide object arrangement unit that arranges a guide object for coordinate detection in advance in a virtual space; a video renderer that receives a video stream and performs video rendering on the input video stream to reproduce a video corresponding to the viewpoint position; an audio renderer that receives an audio stream and performs audio rendering on the input audio stream to reproduce audio corresponding to the viewpoint position; a coordinate detection unit to which a captured image of a display unit reproducing the video is input and which detects a first coordinate of a guide object included in the input captured image; a difference calculation unit that calculates a difference between a second coordinate of the guide object arranged in advance and a first coordinate of the guide object detected by the coordinate detection unit; a delay correction unit that delays the audio reproduced by the audio renderer based on the difference calculated by the difference calculation unit; A video / audio reproducing device comprising:

4. The video / audio playback device of claim 3, characterized in that the delay correction unit converts the difference calculated by the difference calculation unit into a time difference, and if the converted time difference exceeds a predetermined threshold, delays the audio played by the audio renderer based on the time difference.

5. A program for causing a computer to function as the video / audio reproducing device according to any one of claims 1 to 4.

Images