Localization processing apparatus and program

Abstract

To provide a localization processing device and a program capable of converting a localization direction of a sound object in accordance with a display range displayed in a display area when content is reproduced.SOLUTION: An equation generation unit configured to generate, when VR content including video content in a three dimensional virtual space and audio content using object-based sound is reproduced in a real space, a conversion equation for converting a direction in the virtual space into a direction in the real space on the basis of a real display range in a case where the video content is displayed in a two dimensional display region in the real space and a virtual display range of the video content in the virtual space, the virtual display range corresponding to the real display range; SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a localization processing device and a program that perform processing on the localization direction of an audio object during content playback. [Background technology]

[0002] Object-based audio is a well-known audio technology that records audio objects and audio metadata during content creation and renders the content in a format appropriate for the playback environment. In object-based audio, information such as the position and gain of audio objects is described as audio metadata, and a renderer generates a playback signal based on the audio metadata (see, for example, Non-Patent Document 1).

[0003] Object-based audio is being applied to content that uses virtual reality (VR) technology. Furthermore, object-based audio is being expanded and developed for 6DoF (Degrees of Freedom) content, which allows users to view content from various positions and angles (see, for example, Non-Patent Document 2).

[0004] Such VR content is often produced so that it can be viewed using a head-mounted display (HMD), which covers the user's field of vision. However, a user may also view VR content using, for example, a small mobile terminal device (such as a smartphone or tablet). In such cases, an audio object may be localized in a direction different from the direction intended when the VR content was produced. A technology for eliminating such deviation in localization direction is disclosed, for example, in Patent Document 1. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] ITU-R Recommendation BS. 2127-1 [Non-patent document 2] ISO / IEC 23090-4 “MPEG-I Immersive Audio” (under standardization) [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2022-188830 Summary of the Invention [Problem to be solved by the invention]

[0007] The technology disclosed in Patent Document 1 uses video information at the time of content creation to perform a process of correcting the localization direction when playing back the content. For this reason, there is a demand for a method of correcting the localization direction when playing back content even when the video information at the time of content creation is not available.

[0008] The present disclosure aims to provide a localization processing device and program that can convert the localization direction of an audio object to match the display range displayed in a display area when playing back content. [Means for solving the problem]

[0009] A localization processing device according to one embodiment of the present disclosure includes: an equation generation unit that generates a conversion equation for converting a direction in the virtual space into a direction in the real space based on an actual display range when VR (Virtual Reality) content having video content in a three-dimensional virtual space and audio content using object-based audio is played in a real space, and a virtual display range of the video content in the virtual space, the actual display range corresponding to the actual display range; and a conversion unit that converts the virtual localization direction in the virtual space of an audio object included in the audio content into an actual localization direction in the real space based on the conversion equation.

[0010] A program according to one embodiment of the present disclosure causes a computer to execute the following processes when playing VR (Virtual Reality) content having video content in a three-dimensional virtual space and audio content using object-based audio in a real space: a process of generating a conversion formula for converting a direction in the virtual space to a direction in the real space based on an actual display range when the video content is displayed in a two-dimensional display area in the real space and a virtual display range of the video content in the virtual space, the virtual display range corresponding to the actual display range; and a process of converting the virtual positioning direction in the virtual space of an audio object included in the audio content into an actual positioning direction in the real space based on the conversion formula. [Effects of the Invention]

[0011] According to the present disclosure, when playing back content, it is possible to change the localization direction of an audio object in accordance with the display range displayed in the display area. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a content playback system including a localization processing device. [Figure 2]Conceptual diagram showing the state of VR content viewing in the content playback system [Figure 3] A block diagram showing an example of the configuration of a localization processing device. [Figure 4] A diagram for explaining azimuth angles in virtual space and real space. [Figure 5] A diagram for explaining the elevation angle in virtual space and real space. [Figure 6] FIG. 1 is a diagram for explaining a virtual display angle of view, which is the angle of view of a virtual display range in the horizontal direction, and an actual display angle of view, which is the angle of view of an actual display range in the horizontal direction. [Figure 7] FIG. 1 is a diagram for explaining a virtual display angle of view, which is the angle of view of a virtual display range in the vertical direction, and an actual display angle of view, which is the angle of view of an actual display range in the vertical direction. [Figure 8] Flowchart for explaining an example of the operation of the localization processing device [Figure 9] FIG. 10 is a diagram showing the result of converting an azimuth angle using the conversion formula of the first embodiment. [Figure 10] FIG. 10 is a diagram showing the results of converting the elevation angle using the conversion formula of the first embodiment. [Figure 11] FIG. 10 is a diagram showing the result of converting the azimuth angle using the conversion formula of the second embodiment. [Figure 12] FIG. 10 is a diagram showing the results of converting the elevation angle using the conversion formula of the second embodiment. [Figure 13] FIG. 10 is a diagram showing the result of converting the azimuth angle using the conversion formula of the third embodiment. [Figure 14] FIG. 10 is a diagram showing the result of converting the elevation angle using the conversion formula of the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0014] <Configuration of a content playback system including a localization processing device> 1 is a diagram illustrating an example of the configuration of a content playback system including a localization processing device. As shown in FIG. 1, a localization processing device 10 according to an embodiment of the present disclosure is included in a content playback system 100. In addition to the localization processing device 10, the content playback system 100 also includes a playback device 20 and a display device 30.

[0015] The playback device 20 is a device that plays back VR content, which includes video content and audio content.

[0016] In this disclosure, the video content is assumed to be content including video objects in a three-dimensional virtual space. In particular, the video content is assumed to be video content (3DoF content or 6DoF content) that allows the viewpoint to be moved within the virtual space. Furthermore, the audio content is assumed to be audio content using object-based audio, which allows the localization direction of an audio object in real space to be changed according to the position of the viewpoint in the virtual space. In this disclosure, the real space is the space in which the user's body and the display device 30 exist, and is different from the virtual space in which only the user's viewpoint can move. In this disclosure, the localization direction refers to the direction from which the audio is heard relative to the user. Note that the audio object refers to the sound material that constitutes the audio content. Audio objects include commentary, commentary, dialogue, music, background sounds, sound effects, etc. The audio content includes one or more audio objects.

[0017] The playback device 20 can appropriately employ any known technology as a method for playing back the video content and audio content of the VR content.

[0018] When playing back VR content in real space, the playback device 20 can adjust the display range of the video content to match the size of the display area of ​​the display device 30. Playing back VR content in real space means displaying video content having a three-dimensional virtual space on the display device 30 having a two-dimensional display area, and playing back audio content via the localization processing device 10.

[0019] The playback device 20 has an operation unit, and can adjust the display range of the video content to be displayed in the display area of ​​the display device 30 based on an operation input by the user via the operation unit.

[0020] The playback device 20 outputs a video signal to the display device 30 and an audio signal to the localization processing device 10 when playing back VR content.

[0021] Furthermore, the playback device 20 outputs information about the display range of the video content in the virtual space that is aligned with the display area of ​​the display device 30 to the localization processing device 10. In the following description, the information about the display range of the video content in the virtual space that is aligned with the display area of ​​the display device 30 may be referred to as a virtual display range.

[0022] Furthermore, the playback device 20 outputs information about the position of the user's viewpoint in the virtual space when the video content is played back to the localization processing device 10. In the following description, the information about the position of the user's viewpoint in the virtual space may be referred to as a virtual viewpoint position.

[0023] Furthermore, the playback device 20 outputs audio metadata included in the audio content in association with the audio signal to the localization processing device 10. The audio metadata includes information such as the localization direction and gain of the audio object in the virtual space.

[0024] The display device 30 displays video content based on the video signal. In the present disclosure, the display device 30 is assumed to be a mobile terminal device such as a smartphone or a tablet terminal. The display device 30 outputs information regarding the size of its own display area (screen size) to the playback device 20 and the localization processing device 10. In the following description, information regarding the screen size may be simply referred to as the screen size.

[0025] Furthermore, the display device 30 detects information relating to the distance between itself and the user's viewpoint in real space using a sensor or the like, and outputs the information to the localization processing device 10. In the following description, the information relating to the distance between the display device 30 and the user's viewpoint may be referred to as a viewing distance. As the viewing distance, instead of a detected value of the distance between the display device 30 and the user, a distance (e.g., 30 cm) that is generally predicted to be away from the user when holding the display device 30 may be set in advance as a fixed value.

[0026] Based on the virtual display range, virtual viewpoint position, and audio metadata received from the playback device 20, and the screen size and viewing distance received from the display device 30, the localization processing device 10 executes a conversion process to convert the localization direction of each audio object in the virtual space into a localization direction (in real space) that matches the display range of the video content on the display device 30. The localization processing device 10 also reflects the conversion process on the audio signal received from the playback device 20, and outputs the converted audio signal.

[0027] The converted audio signal output by the localization processing device 10 is reproduced by an audio device such as a multi-channel speaker or headphones, and reaches the user's ears.

[0028] Fig. 2 is a conceptual diagram showing the state when VR content is viewed in the content playback system 100. In Fig. 2, the display range of the display device 30 in the real space is shown as the actual display range. In Fig. 2, the display range of the video content in the virtual space, which corresponds to the actual display range, is shown as the virtual display range.

[0029] FIG. 2 shows an example in which the audio content of the VR content includes audio objects A and B. In FIG. 2, the localization direction of audio object A in the virtual space is a virtual localization direction D. A_virtual In FIG. 2, the localization direction of the sound object B in the virtual space is shown as virtual localization direction D B_virtual In FIG. 2, the localization direction of the sound object A in the real space is shown as the real localization direction D A_real In Figure 2, the localization direction of the sound object B in the real space is shown as the real localization direction D B_real The virtual localization direction refers to the localization direction of the audio object in the virtual space, i.e., the localization direction intended when the VR content was created. The actual localization direction refers to the localization direction of the audio object that matches the display range of the video content when the VR content is played in the real space.

[0030] 2, in the content playback system 100, the localization direction (virtual localization direction) of each audio object in the virtual space is corrected to the localization direction (real localization direction) of each audio object in the real space so that it matches the display range of the video content displayed in the real space on the display device 30. This allows the user to experience the VR content in a state where the audio of the audio object is heard from a direction that matches the display content of the video content.

[0031] 1 shows an example in which the content playback system 100 has the localization processing device 10, the playback device 20, and the display device 30 each configured as an independent unit, but the present disclosure is not limited to this. For example, at least some of the functions of the localization processing device 10, the playback device 20, and the display device 30 may be realized by a single device (computer). The functions of the localization processing device 10, the playback device 20, and the display device 30 may each be realized by hardware such as an independent board, or may be realized by software in which one or more processors execute a predetermined program.

[0032] <Configuration of localization processing device 10> The localization processing device 10 according to an embodiment of the present disclosure will be described in detail below. Fig. 3 is a block diagram showing an example of the configuration of the localization processing device 10. As shown in Fig. 3, the localization processing device 10 includes an equation generation unit 11, a conversion unit 12, and a signal generation unit 13. The localization processing device 10 may be realized by any computing device (computer) or information processing device such as a server, a PC, a tablet terminal, or a smartphone.

[0033] The formula generation unit 11 generates a conversion formula for converting a direction in the virtual space into a direction in the real space based on the real display range and the virtual display range. The real display range is the display range when the video content of the VR content is displayed in the two-dimensional display area of ​​the display device 30 in the real space, as shown in FIG. 2. The virtual display range is the display range corresponding to the real display range, and is the display range of the virtual space. In other words, the virtual display range means the range corresponding to the real display range of the entire display range of the video content in the video content having a virtual space.

[0034] The equation generating unit 11 generates a conversion equation for each of the azimuth angle and the elevation angle.

[0035] Here, the azimuth angle and the elevation angle will be described. Fig. 4 is a diagram for explaining the azimuth angle in each of the virtual space and the real space. Fig. 5 is a diagram for explaining the elevation angle in each of the virtual space and the real space.

[0036] FIG. 4A shows a virtual viewpoint position, which is the position of a viewpoint in a virtual space, and an arbitrary object O in the virtual space as seen from the virtual viewpoint position. i (i is a positive integer) and are arranged on a horizontal plane (xy plane). In FIG. 4A, an arbitrary object O seen from a virtual viewpoint in the virtual space is shown. i The azimuth angle indicating the direction of i_virtual It shows:

[0037] FIG. 4B shows a real viewpoint position, which is the position of the viewpoint in real space, and an object O seen from the real viewpoint position in real space. i The azimuth angle indicating the direction of i_real It shows:

[0038] FIG. 5A shows a virtual viewpoint position, which is the position of a viewpoint in a virtual space, and an arbitrary object O in the virtual space as seen from the virtual viewpoint position. i 5A shows the object O as seen from the virtual viewpoint. i The elevation angle indicating the direction of i_virtual It shows:

[0039] FIG. 5B shows a real viewpoint position, which is the position of the viewpoint in real space, and an elevation angle θ , which indicates the direction of an arbitrary object seen from the real viewpoint position in real space. i_real It shows:

[0040] The coordinates of the virtual viewpoint are (u x ,u y ,u z ), and the coordinates of any object in virtual space are (x i ,y i ,z i ), then φ i_virtual and θ i_virtualcan be expressed by the following equation (1).

number

[0041] Returning to the description of the equation generating unit 11, the equation generating unit 11 generates a conversion equation for converting a direction in the virtual space into a direction in the real space, based on the real display range and the virtual display range, as described above.

[0042] More specifically, the equation generating unit 11 calculates an azimuth angle φ in the virtual space based on the viewpoint at the time of reproducing the video content. _virtual and elevation angle θ _virtual and the azimuth angle φ in real space, respectively. _real and elevation angle θ _real In the following description, of the conversion equations generated by equation generation unit 11, the equation for converting the azimuth angle is referred to as function f, and the equation for converting the elevation angle is referred to as function g. That is, equation generation unit 11 generates functions f and g that satisfy the following equations (2) and (3).

number

number

[0043] The formula generating unit 11 generates functions f and g as follows: i is located behind the user at the virtual viewpoint, in other words, |φ _viatual |>π / 2, or |θ _viatual If |>π / 2, then the object O iFunctions f and g are generated so that the actual localization direction of the object is the same as the virtual localization direction. This is because such an object was assumed not to be in the user's field of view at the time of production, and it is clear that it will not be in the user's field of view when the VR content is played back in the real space, so making the virtual localization direction and the actual localization direction the same will allow playback to be in line with the intention of the content creator at the time of production.

[0044] The equation generating unit 11 calculates the equation for -π / 2≦φ. _viatual ≦π / 2, or -π / 2≦θ _viatual If ≦π / 2, the functions f and g are set to monotonically increasing functions. This is because, when multiple sound objects are taken into consideration, by setting the functions f and g to monotonically increasing functions, the order of the sound objects does not change even if the actual localization direction of each sound object changes from the virtual localization direction.

[0045] For example, the formula generator 11 calculates the formula for -π / 2≦φ _viatual ≦π / 2, or -π / 2≦θ _viatual Functions f and g in the range ≦π / 2 are generated as N-th degree polynomials as shown in the following equations (4) and (5), where N is an arbitrary odd number and may be selected from, for example, 5, 7, or 9.

number

number

[0046] a n and b n is a predetermined coefficient. n and b n The angle of view Δφ of the virtual display range seen from the virtual viewpoint position C , Δθ C and the angle of view Δφ of the actual display range as seen from the actual viewpoint position D , Δθ DThis allows an appropriate conversion formula to be generated depending on the size of the display area displayed on the display device 30 (i.e., screen size) and the position of the user (viewpoint) in the virtual space.

[0047] FIG. 6 shows the virtual display angle of view (azimuth angle) Δφ, which is the angle of view of the virtual display range in the horizontal direction. C and the actual display angle of view (azimuth angle) Δφ, which is the angle of view of the actual display range in the horizontal direction D 6A is a diagram for explaining the virtual display angle of view Δφ. C 6B shows the actual display angle of view Δφ D This shows:

[0048] FIG. 7 shows the virtual display angle of view (elevation angle) Δθ, which is the angle of view of the virtual display range in the vertical direction. C and the actual display angle of view (elevation angle) Δθ, which is the angle of view of the actual display range in the vertical direction. D 7A is a diagram for explaining the virtual display angle of view Δθ C 7B shows the actual display angle of view Δθ D This shows:

[0049] Virtual display angle of view Δφ C and Δθ C is determined by the display range (virtual display range) of the video content in the virtual space. The virtual display range changes, for example, every time the user moves the viewpoint or changes direction while the VR content is being played. The formula generation unit 11 constantly acquires the virtual display range from the playback device 20 while the VR content is being played, thereby determining the virtual display angle of view Δφ C and Δθ C It should be noted that the playback device 20 can obtain the virtual display angle of view Δφ C and Δθ C Alternatively, the information about the above may be output directly to the formula generation unit 11.

[0050] As shown in FIGS. 6B and 7B, the actual display angle of view Δφ D and Δθ Dis determined by the screen size (width w and height h), which is the size of the display area of ​​display device 30 on which video content is displayed in real space, and the viewing distance v, which is the distance between display device 30 and the user (real viewpoint position). Equation generation unit 11 acquires the screen size, which indicates the size of the display area, and the viewing distance from display device 30.

[0051] At this time, the actual display angle of view Δφ D and Δθ D can be calculated using the following equations (6) and (7).

number

number

[0052] The equation generating unit 11 calculates the coefficient a n azimuth angle φ in virtual space _virtual and the azimuth angle φ in real space _real The set Φ is determined by the least squares method with monotonic constraints using a set Φ of values ​​that can be taken by . The set Φ is expressed by, for example, the following equation (8).

number

[0053] Similarly, the equation generator 11 calculates the coefficient b n azimuth angle θ in virtual space _virtual and the azimuth angle θ in real space _real The set Θ is determined by the least squares method with monotonicity constraints using a set Θ of values ​​that can be taken by . The set Θ is expressed by the following equation (9), for example.

number

[0054] To estimate monotonically increasing, if n is an even number, n ,b n is estimated to be a sufficiently small value, so a n ,b n= 0. This makes the functions f and g Nth-order polynomials with only odd-order coefficients.

[0055] As a result of the above, the expression generation unit 11 generates the following functions f and g.

number

number

[0056] In equations (10) and (11), N' is the largest natural number that satisfies 2N'-1≦N.

[0057] The functions f and g, which are the conversion equations generated in this way, are functions that convert any direction (azimuth angle or elevation angle) in virtual space into a specific direction (azimuth angle or elevation angle) in real space. The equation generation unit 11 outputs the generated functions f and g to the conversion unit 12 as conversion equations.

[0058] The conversion unit 12 converts the virtual localization direction in the virtual space into the real localization direction in the real space for each audio object included in the audio content using the functions f and g. i The virtual azimuth angle that indicates the virtual orientation direction of i_virtual , virtual elevation angle θ i_virtual , the audio object O i The actual azimuth angle that indicates the actual orientation direction of i_real , actual elevation angle θ i_virtual Then, the following equations (12) and (13) hold.

number

number

[0059] In this way, the conversion unit 12 converts the virtual localization direction into the actual localization direction for each sound object. The conversion unit 12 outputs information about the actual localization direction (azimuth angle and elevation angle) for each sound object to the signal generation unit 13.

[0060] The signal generation unit 13 generates an output audio signal in which the localization direction of each audio object in the audio signal is corrected to the actual localization direction based on the actual localization direction (azimuth and elevation) of each audio object obtained from the conversion unit 12 and the audio signal obtained from the playback device 20.

[0061] The signal generation unit 13 outputs the output audio signal to an audio output device such as a speaker, headphones, earphones, etc. In this way, the localization processing device 10 can provide the user with an experience of hearing audio from the actual localization direction corrected for each audio object in real space.

[0062] <Example of operation of the localization processing device 10> FIG. 8 is a flowchart for explaining an example of the operation of the localization processing device 10.

[0063] In step S1, the localization processing device 10 acquires various pieces of information from the playback device 20 and the display device 30 (see FIG. 3).

[0064] In step S2, the localization processing device 10 determines whether the viewpoint has moved during playback of the VR content, based on the virtual viewpoint position acquired from the playback device 20. If it is determined that the viewpoint has moved (step S2: YES), the localization processing device 10 proceeds to step S3. If it is determined that the viewpoint has not moved, or if playback of the VR content has just started (step S2: NO), the localization processing device 10 proceeds to step S4.

[0065] In step S3, the localization processing device 10 generates a new transformation formula (functions f, g) based on the viewpoint after movement.

[0066] In step S4, the localization processing device 10 converts the localization direction for each audio object using the latest conversion formula, so that the localization direction for each audio object matches the playback environment in the real space (the playback environment in the real space where the video content is displayed on the display device 30).

[0067] In step S5, the localization processing device 10 generates and outputs an output audio signal that reflects the converted localization direction in the audio signal acquired from the playback device 20. This allows the user to have an audio experience that matches the playback environment in the real space (the playback environment in the real space where the video content is displayed on the display device 30).

[0068] In step S6, the localization processing device 10 determines whether the VR content being played has ended. If it is determined that the VR content has ended (step S6: YES), the localization processing device 10 ends its operation. If it is determined that the VR content has not ended (step S6: NO), the localization processing device 10 returns the process to step S1.

[0069] By performing such an operation, the localization processing device 10 can correct the actual localization direction of each audio object to match the user's content reproduction environment (the environment in which the video content is displayed on the display device 30).

[0070] <Example> In the following, an embodiment will be described in which the formula generating unit 11 of the localization processing device 10 generates conversion formulas for various parameters, and conversion (correction) of the localization direction for each audio object is performed by simulation.

[0071] Example 1 In Example 1, the width w of the display area of ​​the display device 30 is 0.15 [m], the height h is 0.08 [m], the viewing distance (the distance between the display device 30 and the user) v is 0.4 [m], and the virtual display angle of view (azimuth angle) Δφ C is π / 3, virtual display angle of view (elevation angle) Δθ D This is an example in which the number of terms in the conversion formula, N, is 9 (N'=5).

[0072] At this time, the coefficient a of the conversion formula of the azimuth angle φ calculated by the formula generating unit 11 of the localization processing device 10 is n The coefficient b of the conversion formula for the elevation angle θ is shown in Table 1. n is shown in Table 2. [Table 1] [Table 2]

[0073] 9 shows the result of converting the azimuth angle φ∈[-π,π] using the conversion formula of Example 1, and Fig. 10 shows the result of converting the elevation angle θ∈[-π / 2,π / 2]. Fig. 10A shows the result of converting the front side as seen from the user, and Fig. 10B shows the result of converting the rear side.

[0074] In the azimuth angle conversion results shown in Figure 9, the azimuth angle is -Δφ C More than Δφ C In the following range, the azimuth angle is converted to match the actual display angle of view (screen size), which is significantly smaller than the virtual display angle of view. When the localization direction (azimuth angle) of the audio object is converted (corrected) using such a conversion formula, the audio object displayed in the display area of ​​the display device 30 is converted (corrected) to match the actual display angle of view, which is significantly smaller than the virtual display angle of view, in real space. D More than Δφ D This allows the user to experience the sound of the audio object displayed in the display area of ​​the display device 30 as if it were coming from a direction within the range of the display area in terms of azimuth.

[0075] And the azimuth angle Δφ C greater than or equal to π / 2 and less than or equal to -Δφ C In the range of −π / 2 or more, that is, the range in front of the user in the virtual space and not displayed in the display area of ​​the display device 30, the virtual display angle of view Δφ C The actual display angle of view Δφ D Since it is significantly larger than the azimuth angle -Δφ,C More than Δφ C The amount of change due to the conversion of the azimuth angle is suddenly large compared to the range below. When the localization direction (azimuth angle) of an audio object is converted (corrected) using such a conversion formula, the user can experience the audio of an audio object that is not displayed in the display area of ​​the display device 30 and that should be in front of the user as if it were coming from outside the display area and in front (i.e., to the side of the display device 30).

[0076] In the range of azimuth angles less than -π / 2 and more than π / 2, i.e., the range behind the user in the virtual space, the input azimuth angle is output as is (identity transformation). If the localization direction (azimuth angle) of an audio object is converted (corrected) using such a transformation formula, an audio object located behind the user in the virtual space will also be localized behind the user in the real space.

[0077] The elevation angle conversion results shown in Figure 10 also show that the elevation angle is -Δθ C More than Δθ C In the following range, the elevation angle is converted to match the actual display angle of view (screen size), which is significantly smaller than the virtual display angle of view. When the localization direction (elevation angle) of the audio object is converted (corrected) using such a conversion formula, the audio object displayed in the display area of ​​the display device 30 is converted (corrected) to match the actual display angle of view, which is significantly smaller than the virtual display angle of view, in real space. D More than Δθ D This allows the user to experience the sound of the audio object displayed in the display area of ​​the display device 30 as if it were coming from a direction within the range of the display area, at an elevation angle.

[0078] And the elevation angle Δθ C greater than or equal to π / 2 and less than or equal to -Δθ C In the range of −π / 2 or more, that is, the range in front of the user in the virtual space and not displayed in the display area of ​​the display device 30, the virtual display angle of view Δθ C The actual display angle of view ΔθD Since it is significantly larger than the elevation angle, the elevation angle -Δθ C More than Δθ C The amount of change due to the conversion of the elevation angle is suddenly large compared to the range below. When the localization direction (elevation angle) of an audio object is converted (corrected) using such a conversion formula, the user can experience the audio of an audio object that is not displayed in the display area of ​​the display device 30 and should be in front of the user as if it is coming from outside the display area and in front (i.e., either above or below the display device 30).

[0079] In the range of elevation angles less than -π / 2 and more than π / 2, i.e., the range behind the user in the virtual space, the input elevation angle is output as is. When the localization direction (elevation angle) of an audio object is converted (corrected) using such a conversion formula, an audio object located behind the user in the virtual space will also be localized behind the user in the real space.

[0080] Example 2 In Example 2, the width w of the display area of ​​the display device 30 is 0.70 [m], the height h is 0.40 [m], the viewing distance (the distance between the display device 30 and the user) v is 0.4 [m], and the virtual display angle of view (azimuth angle) Δφ C is π / 3, virtual display angle of view (elevation angle) Δθ D This is an example in which the number of terms in the conversion formula, N, is 9 (N'=5).

[0081] At this time, the coefficient a of the conversion formula of the azimuth angle φ calculated by the formula generating unit 11 of the localization processing device 10 is n The coefficient b of the conversion formula for the elevation angle θ is shown in Table 3. n is shown in Table 4. [Table 3] [Table 4]

[0082] 11 shows the result of converting the azimuth angle φ∈[-π,π] using the conversion formula of Example 2, and Fig. 12 shows the result of converting the elevation angle θ∈[-π / 2,π / 2]. Fig. 12A shows the result of converting the front side as seen from the user, and Fig. 12B shows the result of converting the rear side.

[0083] In the second embodiment, the virtual display angle of view is the same as that in the first embodiment, but the screen size, that is, the actual display angle of view, is significantly larger than that in the first embodiment. C More than Δφ C The following range and Δθ C The amount of change due to the change in the azimuth angle in the range of π / 2 or more is significantly larger than the amount of change in the same range in Example 1. C In the range of -π / 2 or more, the amount of change in the azimuth angle due to the conversion is gentler than in Example 1. As a result, in the azimuth angle range of -π / 2 to π / 2, the converted azimuth angle in real space has a value similar to that in virtual space. The same is true for the elevation angle shown in FIG.

[0084] In this way, the localization processing device 10 can appropriately set the localization direction (actual localization direction) of each audio object in real space to match the size (screen size) of the display device 30 used by the user to watch video content in real space.

[0085] Example 3 In Example 3, the width w of the display area of ​​the display device 30 is 0.15 [m], the height h is 0.08 [m], the viewing distance (the distance between the display device 30 and the user) v is 0.4 [m], and the virtual display angle of view (azimuth angle) Δφ C is π / 6, virtual display angle of view (elevation angle) Δθ D This is an example in which is π / 12. The number of terms N in the conversion formula is 9 (N'=5).

[0086] At this time, the coefficient a of the conversion formula of the azimuth angle φ calculated by the formula generating unit 11 of the localization processing device 10 is n The coefficient b of the conversion formula for the elevation angle θ is shown in Table 5. n is shown in Table 6. [Table 5] [Table 6]

[0087] 13 shows the result of converting the azimuth angle φ∈[-π,π] using the conversion formula of Example 3, and Fig. 14 shows the result of converting the elevation angle θ∈[-π / 2,π / 2]. Fig. 14A shows the result of converting the front side as seen from the user, and Fig. 14B shows the result of converting the rear side.

[0088] In the third embodiment, the screen size and the viewing distance are the same as those in the first embodiment, but the virtual display angle of view is smaller than that in the first embodiment. C More than Δφ C The difference between the virtual display angle of view and the actual display angle of view in the following range is smaller than that in Example 1. As a result, as shown in FIG. C More than Δφ C In the following range, the amount of change in the azimuth angle according to the conversion formula of Example 3 is larger than the amount of change in Example 1. C More than Δφ C Below and Δθ C In the range of π / 2 or more and less than π / 2, the amount of change in the azimuth angle according to the conversion formula of Example 3 is gentler than the amount of change in Example 1. As a result, in the azimuth angle range from -π / 2 to π / 2, the azimuth angle after conversion in real space has a value similar to the azimuth angle in virtual space. The same is true for the elevation angle shown in FIG.

[0089] In this way, when the viewpoint moves in the virtual space during playback of VR content and the virtual display angle of view changes, the localization processing device 10 regenerates the conversion formula to match the change, so that even in content that allows the viewpoint to move, the localization direction of the audio object in the real space can be suitably corrected. [Industrial Applicability]

[0090] The present disclosure is useful for a localization processing device that corrects the localization direction of an audio object when playing back VR content. [Explanation of symbols]

[0091] 100 Content Playback System 10. Stereotactic Processing Device 11 Expression generator 12 Conversion unit 13 Signal generation unit 20 Playback device 30 Display device

Claims

1. a formula generating unit configured to generate a conversion formula for converting a direction in the virtual space into a direction in the real space based on an actual display range when the video content in a three-dimensional virtual space and the audio content using object-based audio are displayed in a two-dimensional display area in the real space and a virtual display range of the video content in the virtual space, the actual display range corresponding to the actual display range; and a conversion unit that converts a virtual location direction of an audio object included in the audio content in the virtual space into a real location direction of the audio object in the real space based on the conversion formula; A localization processing device comprising:

2. The formula generation unit calculates an azimuth angle φ in the virtual space based on a viewpoint at the time of playing the video content. ＿ｖｉｒｔｕａｌ and elevation angle θ ＿ｖｉｒｔｕａｌ and the azimuth angle φ in the real space. ＿ｒｅａｌ and elevation angle θ ＿ｒｅａｌ and generating the transformation formula for transforming each of the The localization processing device according to claim 1 .

3. The formula generation unit generates the conversion formula as follows: |φ ＿ｖｉｒｔｕａｌ |≦T (T is a predetermined threshold angle) or |θ ＿ｖｉｒｔｕａｌ |≦T is a monotonically increasing function, |φ ＿ｖｉｒｔｕａｌ |>T or |θ ＿ｖｉｒｔｕａｌ |> In T, generate it as the identity function, 3. The actual positioning direction determining device according to claim 2.

4. the formula generation unit generates a conversion formula f shown in formula (1) and a conversion formula g shown in formula (2) as the monotonically increasing function.

4. The actual positioning direction determining device according to claim 3. [Equation 1] [Equation 2] In equations (1) and (2), φ is the azimuth angle, θ is the elevation angle, N is a positive odd number, and a ｎ and b ｎ is a predetermined coefficient.

5. The equation generation unit calculates the coefficient a by a monotone constrained least squares method. ｎ and b ｎ Calculate The localization processing device according to claim 4 .

6. a process for generating a conversion formula for converting a direction in the virtual space into a direction in the real space based on an actual display range when the video content in a three-dimensional virtual space and the audio content using object-based audio are displayed in a two-dimensional display area in the real space and a virtual display range of the video content in the virtual space, the virtual display range corresponding to the actual display range; and a process of converting a virtual localization direction of an audio object included in the audio content in the virtual space into a real localization direction in the real space based on the conversion formula; A program that causes a computer to execute the following.

Images