Placing sounds within content

Abstract

A device includes components for performing the following operations: displaying content within a viewport; displaying a symbol representing a sound source within the viewport; detecting one or more gestures that place the displayed symbol within the displayed content; and disambiguating between a first placement method performed on the symbol and a second placement method performed on the symbol that is different from the first placement method, the first placement method locking the sound source to a position that moves as the displayed content moves within the viewport, and the second placement method locking the sound source to a position that does not move as the displayed content moves within the viewport, wherein the first placement method includes a disambiguating gesture for the symbol relative to the viewport, the disambiguating gesture indicating movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to placing sound within displayed content. Background Art

[0002] Spatial audio allows users to control where sound sources are rendered from. In some cases, users want to add audio sources to the content being displayed. Summary of the Invention

[0003] According to various, but not necessarily all, embodiments, there is provided an apparatus comprising means for:

[0004] Display content within the viewport;

[0005] Display symbols representing sound sources in the viewport;

[0006] detecting one or more gestures that place the displayed symbol within the displayed content;

[0007] disambiguating between a first placement method for symbolic execution that locks the sound source to a position that moves as displayed content moves within a viewport and a second placement method for symbolic execution that is different from the first placement method and that locks the sound source to a position that does not move as displayed content moves within the viewport,

[0008] The first placement method includes a disambiguating gesture for the symbol relative to the viewport, the disambiguating gesture indicating movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

[0009] In some, but not necessarily all, examples, the disambiguation gesture of the first placement method includes dragging the symbol relative to the viewport.

[0010] In some, but not necessarily all, examples, the first placement method includes a previous gesture that dragged the symbol relative to the viewport and displayed content prior to the disambiguating gesture.

[0011] In some, but not necessarily all, examples include components for:

[0012] determining a first position in the viewport, the first position defined by an endpoint of a previous gesture drag symbol relative to the viewport and displayed content; and

[0013] determining a second position in the viewport, the second position being defined by the termination of the corresponding first placement method or the second placement method; wherein a difference between the first position and the second position disambiguates the first placement method and the second placement method

[0014] In some but not necessarily all examples, the first drop method includes a drop method in which the symbol is dragged to content and then dragged with the content, and the second drop method includes a drop method in which the symbol is dragged to a position within the viewport and then maintained at the position within the viewport.

[0015] In some but not necessarily all examples, the first placement method and the second placement method include selecting the symbol using a gesture.

[0016] In some but not necessarily all examples, the first placement method and the second placement method include: an initial gesture for the symbol relative to the viewport and displayed content, the initial gesture indicating movement of the symbol relative to the viewport and displayed content,

[0017] The first placement method additionally includes:

[0018] For a gesture of the symbol relative to the viewport, the gesture indicates movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

[0019] In some but not necessarily all examples, the first placement method includes

[0020] An initial drag movement of the symbol relative to the viewport and displayed content, and subsequent movement of the symbol relative to the viewport, but without movement of the symbol relative to the displayed content.

[0021] In some but not necessarily all examples, the first placement method includes

[0022] An initial drag movement of the symbol relative to the viewport and displayed content, and a subsequent movement of the symbol relative to the viewport to a variable position that tracks the displayed content being moved within the viewport to which the sound source is to be locked.

[0023] In some, but not necessarily all, examples, the displayed content that is tracked and to which the sound source is locked is the first content of the visual scene.

[0024] In some, but not necessarily all, examples, the first content is a majority of the visual scene forming the background.

[0025] In some, but not necessarily all, examples, the first content is a small portion of the visual scene that is a visual object moving within the visual scene.

[0026] In some but not necessarily all examples, the content displayed within the viewport is a subset of the available content, where the subset is defined by the viewport, and where the viewport is defined by a viewpoint and a field of view, where one or both of the displayed content and the viewpoint are time-varying.

[0027] In some, but not necessarily all, examples, the first placement method includes a hold gesture that maintains the displayed symbol in a fixed position relative to the content while the viewport moves and wherein the second placement method includes a hold gesture that maintains the displayed symbol in a fixed position relative to the viewport while the viewport moves.

[0028] In some, but not necessarily all, examples, the second placement method performed on the symbol locks the sound source to the position of the tracking viewport rather than to the displayed content that is moving relative to the viewport.

[0029] In some, but not necessarily all, examples, the second placement method includes an initial drag movement of the symbol relative to the viewport and displayed content, and no subsequent movement of the symbol relative to the viewport within a threshold time period.

[0030] In some but not necessarily all examples, the viewport is a touch display, and the symbol is an icon displayed on the touch display, and one or more gestures that place the displayed symbol within the displayed content are touch-based gestures.

[0031] In some, but not necessarily all, examples, the icons are provided in a menu that is displayed on the touchscreen display.

[0032] In some, but not necessarily all, examples, the sound source and content are recorded simultaneously.

[0033] According to various, but not necessarily all, embodiments, there is provided a method comprising:

[0034] Display content within the viewport;

[0035] Display symbols representing sound sources in the viewport;

[0036] detecting one or more gestures that place the displayed symbol within the displayed content;

[0037] disambiguating between a first placement method for symbolic execution that locks the sound source to a position that moves as displayed content moves within a viewport and a second placement method for symbolic execution that is different from the first placement method and that locks the sound source to a position that does not move as displayed content moves within the viewport,

[0038] The first placement method includes a disambiguating gesture for the symbol relative to the viewport, the disambiguating gesture indicating movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

[0039] According to various but not necessarily all embodiments, there is provided a computer program comprising program instructions for causing an apparatus to at least perform the following operations:

[0040] Display content within the viewport;

[0041] Display symbols representing sound sources within the viewport; and

[0042] detecting one or more gestures that place the displayed symbol within the displayed content;

[0043] disambiguating between a first placement method for symbolic execution that locks the sound source to a position that moves as displayed content moves within a viewport and a second placement method for symbolic execution that is different from the first placement method and that locks the sound source to a position that does not move as displayed content moves within the viewport,

[0044] The first placement method includes a disambiguating gesture for the symbol relative to the viewport, the disambiguating gesture indicating movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

[0045] According to various, but not necessarily all, embodiments, there is provided a human-machine interface comprising means for:

[0046] Display content within the viewport;

[0047] Display symbols representing sound sources in the viewport;

[0048] detecting one or more gestures that place the displayed symbol within the displayed content;

[0049] disambiguating between a first placement method for symbolic execution that locks the sound source to a position that moves as displayed content moves within a viewport and a second placement method for symbolic execution that is different from the first placement method and that locks the sound source to a position that does not move as displayed content moves within the viewport,

[0050] The first placement method includes a disambiguating gesture for the symbol relative to the viewport, the disambiguating gesture indicating movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

[0051] Examples as described in the following description are provided according to various, but not necessarily all, embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Some example embodiments will now be described with reference to the accompanying drawings, in which:

[0053] Figure 1 An example embodiment of the subject matter described herein is shown;

[0054] FIG2 illustrates another example embodiment of the subject matter described herein;

[0055] FIG3 illustrates another example embodiment of the subject matter described herein;

[0056] Figure 4 Another example embodiment of the subject matter described herein is shown;

[0057] Figure 5 Another example embodiment of the subject matter described herein is shown;

[0058] FIG6 illustrates another example embodiment of the subject matter described herein;

[0059] FIG7 illustrates another example embodiment of the subject matter described herein;

[0060] FIG8 illustrates another example embodiment of the subject matter described herein;

[0061] FIG9 illustrates another example embodiment of the subject matter described herein;

[0062] FIG10 illustrates another example embodiment of the subject matter described herein;

[0063] FIG11 illustrates another example embodiment of the subject matter described herein;

[0064] FIG12 illustrates another example embodiment of the subject matter described herein;

[0065] FIG13 illustrates another example embodiment of the subject matter described herein;

[0066] Figure 14 Another example embodiment of the subject matter described herein is shown;

[0067] FIG15 illustrates another example embodiment of the subject matter described herein;

[0068] Figure 16 Another example embodiment of the subject matter described herein is shown;

[0069] FIG17 shows another example embodiment of the subject matter described herein. DETAILED DESCRIPTION

[0070] Figure 1 An example of an apparatus 10 for controlling the addition of a sound source 50 to displayed content 20 is illustrated.

[0071] The apparatus 10 includes components for the following operations:

[0072] Display content 20 within viewport 30;

[0073] Displaying a symbol 40 representing a sound source 50 within the viewport 30;

[0074] detecting one or more gestures 60 that place the displayed symbol 40 within the displayed content 20;

[0075] Eliminate ambiguity between a first placement method performed on the symbol 40 and a second placement method performed on the symbol 40 that is different from the first placement method, wherein the first placement method locks the sound source 50 to a position that moves as the displayed content 20 moves within the viewport 30, and the second placement method locks the sound source 50 to a position that does not move as the displayed content 20 moves within the viewport 30.

[0076] The first placement method includes a gesture 60 for the symbol 40 relative to the viewport 30 that indicates movement of the symbol 40 relative to the viewport 30 that disambiguates the first placement method from the second placement method.

[0077] In the illustrated example, the apparatus 10 comprises

[0078] A display component 12, the display component 12 is used to display the content 20 in the viewport 30, and is used to display the symbol 40 representing the sound source 50 in the viewport 30;

[0079] a detection component 14 for detecting one or more gestures 60 , the one or more gestures 60 placing the displayed symbol 40 within the displayed content 20 ;

[0080] a disambiguation component 16 for disambiguating between a first placement method performed on the symbol 40 and a second placement method performed on the symbol 40; and

[0081] The spatial audio component 18 is configured to lock the sound source 50 to a position that moves as the displayed content 20 moves within the viewport 30 or to lock the sound source 50 to a position that does not move as the displayed content 20 moves within the viewport 30 depending on the disambiguation.

[0082] The first placement method includes a gesture 60 for the symbol 40 relative to the viewport 30 that indicates movement of the symbol 40 relative to the viewport 30 that disambiguates the first placement method from the second placement method.

[0083] Display component 12 may be any device for controlling or providing visual output to a user. In some, but not necessarily all, examples, apparatus 10 is a handheld apparatus and display component 12 is a display screen of apparatus 10. In some, but not necessarily all, examples, display component 12 projects light into the user's eyes, such as via a near-eye display or a head-mounted display, or is an output interface for controlling the projection of light into the user's eyes or a head-mounted display.

[0084] Detection component 14 can be any device for detecting one or more gestures 60. A gesture can be any movement of a user's body. In some examples, one or more gestures are touch gestures in which a user touches a touch-sensitive screen (e.g., a touch-sensitive display screen) with a finger of a pointing device. In some examples, one or more gestures are three-dimensional gestures in which a user moves his or her body relative to device 10. Three-dimensional gestures can be detected in a variety of ways, for example, using multiple detectors to detect spatial changes in a projected electric field, ultrasound field, or light field.

[0085] The spatial audio component 18 is any device capable of controlling the perceived location of a sound source when presented to a user. The sound source can be modeled as a virtual speaker, which is positioned by controlling, for example, the timing and amplitude differences between the sound perceived at the user's left and right ears. Reverberation can also be used to externalize the sound to the user wearing the headphones. The sound can be presented using a suitable multi-speaker setup (such as surround sound speakers) or using headphones.

[0086] The viewport 30 is a display area visible to the user. In the case where the device 10 uses a display screen as the display component 12, the viewport 30 is the display screen.

[0087] Display content 20 is content displayed in the display area. In some, but not necessarily all, examples, the content is video content. In some, but not necessarily all, examples, the content displayed in viewport 30 at a particular time is a subset of the content available for display in viewport 30 at that particular time. For example, the field of view of the available content may be larger than the field of view of the displayed content. This may occur, for example, if the viewport displays only a portion of a panoramic image, or if viewport 30 displays mediated reality content. In a first-person mediated reality, the user's real viewpoint (position and / or orientation) determines the virtual user's viewpoint within the virtual space, thereby changing the virtual scene (the displayed content). In some examples, the virtual space may extend to 360° horizontally and to 180° vertically.

[0088] In some, but not necessarily all, examples, the sound source 50 and the content are recorded simultaneously.

[0089] In some examples, apparatus 10 is configured to be able to add spatial audio effects (sound source 50 ) to captured content, such as captured video.

[0090] In some examples, displayed content may have associated sounds that are presented while the content is displayed.

[0091] In some, but not necessarily all, examples, device 10 is a rendering device. For example, it can be used to edit content to add spatial audio effects.

[0092] Figures 2A to 2G Examples of viewports 30 are shown. Content 20 is displayed within viewport 30. Content 20 is moving within viewport 30. In these examples, but not necessarily all, the content is Figure 2A and Figure 2B is stationary, and then Figure 2C 、 Figure 2D 、 Figure 2E Move left (relative to viewport 30) in the Figure 2F Move right (relative to viewport 30) in the Figure 2G Move left (relative to viewport 30). Figures 2A to 2G An example of a first placement method 101 is shown.

[0093] exist Figure 2A , a symbol 40 representing a sound source 50 is displayed within the viewport 30. A user 70 performs a selection gesture 60 on the symbol 40.

[0094] Figure 2B A user 70 is illustrated performing a placement gesture 60. The gesture 60 is relative to the viewport 30 and displayed content 20 and indicates movement of the symbol 40 relative to the viewport 30 and displayed content 20.

[0095] For example, drop gesture 60 may be a dragging movement of symbol 40 relative to viewport 30 and displayed content 20 to symbol location 61 in viewport 30 (and displayed content 20 ).

[0096] Figure 2C and Figure 2D User 70 is shown continuing placement gesture 60. This continued placement gesture 60 includes movement of the symbol relative to viewport 30, but without substantial movement of symbol 40 relative to displayed content 20. Symbol position 61 changes relative to viewport 30, but does not change or change significantly relative to content 20. Symbol position 61 follows the same portion of content 20.

[0097] The symbol 40 has a variable position 61 that tracks the displayed content 20 that is moving within the viewport 30 to which the sound source 50 is to be locked.

[0098] For example, a continue drop gesture 60 may be a dragging movement of the symbol 40 relative to the viewport 30, following the displayed content 20. There is no noticeable movement relative to the displayed content 20.

[0099] The displayed content 20 that is tracked and to which the sound source 50 is to be locked is the first content of the visual scene.

[0100] In some, but not necessarily all, examples, the first content is a majority of the visual scene forming the background.

[0101] In some, but not necessarily all, examples, the first content is a small portion of the visual scene that is a visual object moving within the visual scene.

[0102] Figure 2E 、 Figure 2F and Figure 2G The illustration shows that after the placement gesture 60 is terminated, the sound source 50 is locked to a position 51 that moves as the displayed content 20 moves within the viewport 30. The term "locked" means that the sound source 50 is fixed to the position 51 without the need to continue the placement gesture 60. The lock is not necessarily permanent and can be time-limited, for example.

[0103] Although the sound sources 50 are illustrated as visual elements in the viewport 30, this is optional. In some examples, the locations 51 of the sound sources 50 are not visually marked. In some examples, where the locations 51 of the sound sources 50 are visually marked using a visual element, the visual element may be a symbol 40 or a modified version of the symbol. In some examples, the user may select to have the locations of one or more sound sources 50 visually indicated using a symbol 40. The placement methods 101 and 102 may then be performed for any of the displayed symbols 40.

[0104] Figures 3A to 3G An example of a viewport 30 is shown. Content 20 is displayed within the viewport 30. Content 20 is moving within the viewport 30. In these examples, but not necessarily all, the content is Figure 3A and Figure 3B is stationary, and then Figure 3C 、 Figure 3D 、 Figure 3E Move left (relative to viewport 30) in the Figure 3F 、 Figure 3G Move right (relative to viewport 30). Figures 3A to 3G An example is used to illustrate the second placement method 102 .

[0105] To facilitate comparison of the first placement method 101 and the second placement method 102, the same viewport 30, the same mobile display content 20, and the same symbol 40 are used. However, it should be understood that this is merely for ease of comparison, and one or more of these features may differ between the methods.

[0106] exist Figure 3A , a symbol 40 representing a sound source 50 is displayed within the viewport 30. A user 70 performs a selection gesture 60 on the symbol 40.

[0107] Figure 3B A user 70 is illustrated performing a placement gesture 60. The gesture is relative to the viewport 30 and displayed content 20 and indicates movement of the symbol 40 relative to the viewport 30 and displayed content 20.

[0108] For example, drop gesture 60 may be a dragging movement of the symbol relative to viewport 30 and displayed content 20 to a symbol location 61 in viewport 30 (and displayed content 20 ).

[0109] Figure 3C and Figure 3D User 70 is illustrated continuing drop gesture 60. This continued drop gesture 60 includes movement of the symbol relative to displayed content 20, but without substantial movement of symbol 40 relative to viewport 30. Symbol position 61 changes relative to displayed content 20, but does not change or changes significantly relative to viewport 30.

[0110] The symbol 40 has a rest position 61 within the viewport 30 to which the sound source 50 will be locked. This position 61 does not track the displayed content 20 as it moves within the viewport 30.

[0111] For example, the continue drop gesture 60 may be a stationary gesture with no dragging movement of the symbol 40 relative to the viewport 30. There is no noticeable movement relative to the viewport 30.

[0112] The drop gesture 60 includes an initial drag movement of the symbol 40 relative to the viewport 30 and the displayed content 20 ( Figure 3B ) and no subsequent movement of the symbol 40 relative to the viewport 30 within the threshold time period ( Figure 3C 、 3D ). For example, the threshold time period may be 1 or 2 seconds.

[0113] Figure 3E 、 Figure 3F and Figure 3GThe illustration shows that after the placement gesture 60 is terminated, the sound source 50 is locked to a position 51, which does not move as the displayed content 20 moves within the viewport 30. The term "locked" means that the sound source 50 is fixed to the position 51 without the need to continue the placement gesture 60. The lock is not necessarily permanent and can be time-limited, for example.

[0114] Although the sound sources 50 are shown as visual elements in the viewport 30, this is optional. In some examples, the location 51 of the sound source 50 is not visually marked. In some examples, where the location 51 of the sound source 50 is visually marked using a visual element, the visual element may be a symbol 40 or a modified version of the symbol. In some examples, the user may select to have the location of one or more sound sources 50 visually indicated using a symbol 40. The placement methods 101 and 102 may then be performed for any of the displayed symbols 40.

[0115] Comparing the first placement method 101 with the second placement method 102, the first placement method 101 further includes a gesture 60 for the symbol 40 relative to the viewport 30, the gesture 60 indicating the movement of the symbol 40 relative to the viewport 30. The gesture 60 ( Figure 2C 、 Figure 2D ) disambiguates the first placement method 101 from the second placement method 102.

[0116] In at least some examples, the first placement method 101 includes a method in which the symbol 40 is dragged to a location 61 within the displayed content 20 ( Figure 2B C) and then the drop method ( Figure 2C 、 Figure 2D ).

[0117] In at least some examples, the second placement method 102 includes a placement method in which the symbol 40 is dragged to a location 61 within the viewport 30 and then held at that location 61 within the viewport 30 ( Figure 3C 、 Figure 3D ).

[0118] For example, the disambiguation process may include:

[0119] Determine a first position 61 in the viewport 30 ( Figure 2B 、 Figure 2C ), the first position 61 is defined by the endpoints of the gesture 60 of dragging the symbol 40 relative to the viewport 30 and the displayed content 20.

[0120] Determine the second position 61 in the viewport 30 ( Figure 2D 、 Figure 3D ), the second position 61 is defined by the termination of the corresponding first placement method 101 or second placement method 102.

[0121] Determine the first position 61 ( Figure 2B 、 Figure 2C ) and the second position 61( Figure 2D 、 Figure 3D ) and uses the difference to eliminate the ambiguity between the first placement method 101 and the second placement method 102.

[0122] In the first placement method 101, the second position 61 is separated from the first position 61 in the viewport 30 by dragging the symbol 40 relative to the viewport 30. In the second placement method 102, the second position 61 is separated from the first position 61 in the content space by holding the symbol 40 in a substantially fixed position relative to the viewport 30.

[0123] Thus, a small position difference in content space (or a large position difference in viewport space) indicates that the first placement method 101 has occurred, and a large position difference in content space (or a small position difference in viewport space) indicates that the second placement method 102 has occurred.

[0124] In some, but not necessarily all, examples, the drag gesture 60 is performed as movement of a contact point on the surface of a touch screen display. In these examples, termination of the placement method 101, 102 occurs when contact is terminated by lifting a user's finger or pointing device.

[0125] In the case where the viewport 30 is a touch-sensitive display, the symbol 40 can be an icon displayed on the touch-sensitive display. One or more gestures 60 that place the displayed symbol 40 within the displayed content are touch-based gestures performed on the icon. The icon can be provided in a menu displayed on the touch-sensitive display as part of a human-machine interface for adding spatial audio effects to visual content.

[0126] In some, but not necessarily all, examples, the displayed content 20 displayed within the viewport 30 is a subset of the available content.

[0127] like Figure 4 and Figure 5 As shown, the subset is defined by a viewport 30 .

[0128] Figure 4 Examples 1, 2, 3, 4, 5, and 6 in Figure 2B 、 Figure 2C 、 Figure 2D 、 Figure 2E 、 Figure 2F 、 Figure 2G Corresponding. Figure 5 Examples 7, 8, 9, 10, 11, and 12 in Figure 3B 、 Figure 3C 、 Figure 3D 、 Figure 3E 、 Figure 3F 、 Figure 3G Corresponding.

[0129] Figure 4 and Figure 5 The relationship of the viewport 30 to the available content ABCDE is shown. The displayed content 20 is the portion of the available content that overlaps the viewport 30. The viewport 30 moves relative to the available content which changes the displayed content 20.

[0130] In Examples 1 through 3 corresponding to the first placement method 101, the position 61 of the gesture-controlled symbol 40 remains static relative to the specific portion (C) of the displayed content 20 and moves within the viewport 30 in the opposite sense and to the same extent as how the viewport moves relative to the available content 20. Detection of this placement method 101 causes the position 51 of the sound source 50 to be locked to the specific portion (C) of the displayed content 20 defined by the placement method 101. In Examples 4D through 4F, the position 51 of the sound source 50 remains static relative to the specific portion (C) of the displayed content 20 and moves within the viewport 30 in the opposite sense and to the same extent as how the viewport moves relative to the available content 20.

[0131] The first placement method 101 locks the sound source 50 to a position 51 that moves with the displayed content 20 rather than the viewport 30 .

[0132] In this example, the first placement method 101 includes a hold gesture 60 that maintains the displayed symbol 40 in a fixed position 61 relative to the content while the viewport 30 moves.

[0133] In Examples 7 through 9, which correspond to the second placement method 102, the position 61 of the gesture-controlled symbol 40 remains static relative to the particular portion (X) of the viewport 30 and moves with the viewport 30. The position 61 of the gesture-controlled symbol 40 moves within the displayed content 20 in the same sense and to the same extent as how the viewport moves relative to the available content 20. Detection of this placement method 102 causes the position 51 of the sound source 50 to be locked to the particular portion (X) of the viewport defined by the placement method 102. In Examples 5D through 5F, the position 51 of the sound source 50 remains static relative to the particular portion (X) of the viewport 30 and moves within the displayed content 20 in the same sense and to the same extent as how the viewport moves relative to the available content 20.

[0134] The second placement method 102 locks the sound source 50 to a position 51 that moves with the viewport 30 rather than the displayed content 20. The sound source 50 tracks the viewport 30 rather than the displayed content 20 that is moving relative to the viewport 30.

[0135] In this example, the second placement method 102 includes a hold gesture 60 that maintains the displayed symbol 40 in a fixed position 61 relative to the viewport while the viewport moves.

[0136] Figures 6A to 6B and 7A to 7B An example is illustrated where the viewport 30 is a touch display screen of a handheld device.

[0137] Figures 6A to 6B An example of a first placement method 101 is illustrated when the viewport 30 is a touch display screen of a handheld device.

[0138] exist Figure 6A , a symbol 40 representing a sound source 50 is displayed within a viewport 30. A user 70 performs a drop gesture 60 on the symbol 40, and optionally a selection gesture 60. The user touches the symbol 40 with a finger or a pointer, and then drags it to a desired location within the displayed content 20 of the viewport 30.

[0139] exist Figure 6B , user 70 continues placement gesture 60. The user continues to touch symbol 40 with a finger or pointer and drag across viewport 30. This continued placement gesture 60 includes movement of the symbol relative to viewport 30, but no substantial movement of symbol 40 relative to displayed content 20. The symbol position changes relative to viewport 30, but does not change or changes significantly relative to content 20. Symbol position 61 follows the same portion of content 20.

[0140] 7A to 7B An example of the second placement method 102 is illustrated when the viewport 30 is a touch display screen of a handheld device.

[0141] exist Figure 7A , a symbol 40 representing a sound source 50 is displayed within a viewport 30. A user 70 performs a drop gesture 60 on the symbol 40 and optionally performs a selection gesture 60. The user touches the symbol 40 with a finger or a pointer and drags it to a desired location within the displayed content 20 of the viewport 30.

[0142] exist Figure 7B , user 70 continues placement gesture 60. The user continues to touch symbol 40 with a finger or pointer while keeping the finger or pointer substantially stationary relative to viewport 30. This continued placement gesture 60 includes movement of the symbol relative to displayed content 20, but without substantial movement of symbol 40 relative to viewport 30. The symbol position changes relative to displayed content 20, but does not change or changes significantly relative to viewport 30.

[0143] Figure 8A and Figure 8BAn example of how the viewport 30 may be moved relative to changing the available content of the displayed content 20 is illustrated.

[0144] In these examples, the viewport is defined by viewpoint 110 and field of view 112 .

[0145] One or both of the displayed content and the viewpoint 110 are time-varying.

[0146] exist Figure 8A In the illustrated example, the viewpoint 110 can change as the location changes. This corresponds to a tracking shot of the available content.

[0147] exist Figure 8B In the illustrated example, the viewpoint 110 can be varied by a change in orientation. This corresponds to a panning shot of the available content.

[0148] It will be understood that in these examples, the second placement method 102 locks the sound source 50 to the viewpoint 110 so that the position 51 of the sound source 50 relative to the viewpoint 110 is constant as the viewpoint 110 changes. The second placement method 102 locks the sound source 50 to the position 51 that moves with the viewport 30 rather than the displayed content 20. The first placement method 101 locks the sound source 50 to the displayed content that remains the same as the viewpoint 110 changes. The first placement method 101 locks the sound source 50 to the position 51 that moves with the displayed content 20 rather than the viewport 30.

[0149] Figure 9A and Figure 9B Some examples of how the viewpoint 110 may be changed are illustrated.

[0150] Figure 9A An example is illustrated in which display 124 used by device 10 is part of handheld device 120 .

[0151] The handheld device 120 includes a display 124 for displaying displayed content 20 and displayed symbols 40 to the user 70. The handheld device 120 can be intentionally moved in the hand of the user 70 in one or more of six degrees of freedom (three for orientation and three for translation) to define different viewpoints 110.

[0152] If the handheld device has a camera mounted on a face opposite the display, it can operate as a viewing video arrangement that enables viewing of a live real scene while displaying one or more visual elements (such as symbol 40) to the user to combine to provide a virtual scene.

[0153] The handheld device may be or may operate as a viewing video arrangement for mediated reality that enables live or recorded video of a real scene or other content to be displayed on the display 124 for viewing by the user, while one or more visual elements (such as the symbol 40) are simultaneously displayed on the display 124 for viewing by the user. The combination of the displayed real scene or other content and the displayed one or more visual elements provides a virtual scene to the user.

[0154] In some, but not necessarily all, examples, device 10 is an image capture device, such as a camera. Displayed content 20 can be content that is simultaneously displayed in the camera viewfinder and recorded as captured content. This can be used, for example, to edit the captured content to add spatial audio effects. In this example, content-locked sound source 50 originates from the same location in the world even when the camera is panning. In this example, viewport-locked sound source 50 follows the movement of the camera and its viewport. Sound source 50 is rendered from the same direction, regardless of where the camera is pointing.

[0155] Figure 9B An example is illustrated in which display 124 used by device 10 is head mounted device 122 .

[0156] The head mounted device 122 includes a display 124 for displaying displayed content 20 and displayed symbols 40 to the user 70. The head mounted device can automatically move when the user's head moves. The head mounted device 122 can move in one or more of six degrees of freedom (three for orientation and three for translation) to define different viewpoints 110.

[0157] The head mounted device 122 can be operated as a see-through arrangement for augmented reality, which enables viewing of a live real scene to provide a virtual scene while displaying one or more visual elements (such as symbol 40) to the user on the display 124. In this case, the visor (if present) is transparent or translucent so that the live real scene can be viewed through the visor.

[0158] The head mounted device 122 can be operated as a viewing arrangement for mediated reality, which enables live or recorded video of a real scene or other content to be displayed by the display 124 for the user to view, while one or more visual elements (such as the symbol 40) are simultaneously displayed by the display for the user to view. The combination of the displayed scene and the displayed one or more visual elements provides the user with a virtual scene. In this case, the visor is opaque and can serve as the display 124.

[0159] Mediated reality refers to the presentation of mediated reality for the purpose of achieving mediated reality for remote users, such as augmented reality or virtual reality. Mediated reality can support one or more of 3DoF, 3DoF+, or 6DoF.

[0160] A user in real space has a real viewpoint defined by a position and an orientation. The position is a three-dimensional position, and the orientation is a three-dimensional orientation.

[0161] In the example of 3DoF mediated reality, the user's real viewpoint (orientation) determines the virtual viewpoint 110 of the virtual user within the virtual space. The user's orientation controls the virtual orientation of the virtual user. There is a correspondence between the user's real orientation and the virtual orientation of the virtual user, such that a change in the real orientation produces the same change in the virtual orientation.

[0162] The virtual orientation of the virtual user combined with the virtual field of view defines a virtual visual scene within the virtual visual space. The virtual visual scene is presented to the user as displayed content 20. In some examples, the virtual visual scene can additionally define a virtual sound scene.

[0163] The virtual visual scene is the portion of the virtual visual space (available content) displayed to the user via displayed content 20. The virtual sound scene is the portion of the virtual sound space presented to the user. The virtual sound space and the virtual visual space correspond to and form a combined virtual space. The virtual sound space and the virtual visual space correspond to viewpoints within the virtual sound space and have equivalent viewpoints within the virtual visual space. In 3DoF mediated reality, changes in the user's position do not change the virtual position or virtual orientation of the virtual user.

[0164] In the examples of 6DoF and 3DoF+ mediated reality, the user's real viewpoint (position and / or orientation) determines the virtual viewpoint (position and / or orientation) within the virtual space of the virtual user. This is as described for 3DoF, with the addition that the rendered virtual sound scene and the displayed virtual visual scene can be changed by moving the user's position. For example, there may be a mapping between the user's position and the virtual position of the virtual user. A change in the user's position will produce a corresponding change in the virtual position of the virtual user. A change in the virtual position of the virtual user changes the rendered sound scene and also changes the rendered visual scene. In 6DoF, the change in position may result from a change in the user's posture and / or translation by the user by walking or other means. In 3DoF+, the change in position may result from a change in the user's posture.

[0165] In some cases, it may be desirable for a portion of the virtual sound space to remain fixed in the virtual space as the virtual viewpoint of the virtual user changes.

[0166] Viewport-locked sounds defined by the placement method 102 are fixed relative to the virtual user's virtual viewpoint 110 so that when the user changes their viewpoint 110 , the position of the sound source 50 relative to the user does not change.

[0167] Content-locked sounds defined by the placement method 101 are fixed in space so that when the user changes their viewpoint 110, the position 51 of the sound source 50 relative to the user also changes.

[0168] 10A to 13E An example of how the apparatus 10 enables the user 70 to indicate whether a sound source 50 (not shown) represented by a symbol 40 added to the virtual sound space is viewport-locked or content-locked is illustrated in detail.

[0169] Figures 10A to 10E and 12A to 12E The figure shows a virtual user with a viewpoint 110 within a virtual space including a sound source 50. As previously described, in first-person perspective mediated reality, there is a correspondence between the virtual user and the user, and a correspondence between the virtual user's viewpoint and the user's viewpoint. The user can change viewpoint 110 by changing the orientation of the user's head while wearing a head-mounted device 122. This changes the virtual viewpoint 110.

[0170] Figures 11A to 11E The displayed content 20 within the viewport 30 is shown. The displayed content 20 corresponds to Figures 10A to 10E The corresponding virtual visual scene defined by the virtual viewpoint 110.

[0171] exist Figures 10A to 11E In the example of FIG. 5 , the gesture 60 defines the sound source 50 to be added to the virtual sound space of the virtual space as a content-locked sound.

[0172] exist Figure 11A In FIG, a user 70 performs an initial gesture 60 to move the symbol 40 representing the sound source 50 to a desired location 61 in the viewport 30. Figure 13A In the example of , the user grabs the symbol 40 and drags it to a position 61 corresponding to a virtual object 71 in the displayed content 20 .

[0173] exist Figures 11B to 11D , the virtual viewpoint 110 of the virtual user changes, changing the displayed content 20 visible in the viewport 30. The gesture 60 moves relative to the virtual viewpoint 110. The symbol 40 remains static relative to the virtual object 71 by the gesture 60.

[0174] exist Figure 11E The gesture ends. The sound source 50 is then locked to the displayed content 20. Figure 13E In the example of , the sound source 50 is locked to the virtual object 71 .

[0175] 13A to 13E The displayed content 20 is shown in the viewport 30. The displayed content 20 is the same as the content displayed by the corresponding 12A to 12E Corresponding to the virtual visual scene defined by the virtual viewpoint 110 in.

[0176] exist 12A to 13E In the example of , gesture 60 defines a sound source 50 to be added to the virtual sound space of the virtual space as a viewport-locked sound.

[0177] exist Figure 13A In FIG, the user performs an initial gesture 60 to move the symbol 40 representing the sound source 50 to a desired position 61 in the viewport 30. Figure 13A In the example of FIG, the user grabs the symbol 40 and drags it to a position 61 in the viewport 30. For comparison purposes, this position corresponds to a virtual object 71 in the displayed content 20.

[0178] exist 13B to 13D , the virtual viewpoint 110 of the virtual user changes, changing the displayed content 20 visible in the viewport 30. The gesture 60 moves with the virtual viewpoint 110, i.e., with the viewport 30 rather than with the displayed content 20. The symbol 40 remains static relative to the viewport 30 / viewpoint 110 by the gesture 60.

[0179] exist Figure 13E The gesture is terminated. The sound source 50 is then locked to the user's viewpoint 110 and viewport 30.

[0180] The sound source 50 can be either content-locked or viewport-locked. A content-locked sound source 50 has a position that moves with the displayed content 20. For example, it can move relative to the user as the visual scene moves relative to the user, or it can follow an object or other portion of the displayed content that moves within the displayed content. A viewport-locked sound has a position that does not move as the displayed content moves within the viewport, but remains static relative to the viewport and the user. For example, it can remain stationary relative to the user while the visual scene changes relative to the user.

[0181] The sounds presented to the user may include only the content-locked sounds, only the viewport-locked sounds, or a combination of the content-locked sounds and the viewport-locked sounds.

[0182] The user can use the described methods and apparatus to control whether sounds are presented as content-locked sounds or viewport-locked sounds.

[0183] The content-locked sound source 50 generated by the placement method 101 and the viewport-locked sound source 50 generated by the placement method 102 may be presented to the user using the same or different audio output devices. Examples of audio output devices include headphones and speakers.

[0184] When presenting content-locked sound using speakers, the sound source 50 remains static relative to the speakers as the virtual user's viewpoint changes as the user's real orientation changes.

[0185] When presenting content-locked sound using headphones, the position and / or orientation of the sound source 50 needs to change relative to the headphones by the same amount as the virtual viewpoint of the virtual user changes. The position and / or orientation of the sound source 50 is tracked inversely to the changes in the virtual viewpoint 110 of the virtual user, so that the position and / or orientation of the sound source 50 remains fixed in the virtual space and does not move with the virtual viewpoint 110 of the virtual user.

[0186] When presenting viewport-locked sound using headphones, the sound source 50 remains static relative to the headphones as the virtual user's viewpoint changes.

[0187] When using speakers to present viewport-locked sound, the position and / or orientation of the sound source 50 needs to change relative to the speakers by the same amount as the virtual viewpoint of the virtual user changes. The position and / or orientation of the sound source 50 tracks the changes in the virtual viewpoint 110 of the virtual user, so that the position and / or orientation of the sound source 50 remains fixed in real space and moves with the virtual viewpoint 110 of the virtual user.

[0188] Channel-based audio, such as surround sound (e.g., 5.1, 7.1, or 22.2 surround sound) or binaural audio, may be used, or scene-based audio including spatial information about the sound field and sound sources may be used.

[0189] Audio content can encode spatial audio as audio objects. Examples include, but are not limited to, MPEG-4 and MPEG SAOC. MPEG SAOC is an example of metadata-assisted spatial audio.

[0190] The audio content may encode spatial audio as audio objects in the form of moving virtual speakers.

[0191] The audio content can encode spatial audio as an audio signal with parametric side information or metadata. For example, the audio signal can be first-order ambisonics (FOA) or its special case B-format, a higher-order ambisonics (HOA) signal, or mid-side stereo. For such audio signals, a synthesis of the audio signal and parametric metadata is used to synthesize an audio scene so as to create the desired spatial perception.

[0192] Parametric metadata can be generated by different techniques. For example, Nokia's Spatial Audio Capture (OZO Audio) or Directional Audio Coding (DirAC) can be used. Both capture the sound field and represent it using parametric metadata. Parametric metadata may include, for example: a directional parameter indicating the direction of each frequency band; a distance parameter indicating the distance of each frequency band; an energy splitting parameter indicating the ratio of diffuse to total energy per frequency band. Each time-frequency tile can be considered as a sound source, with the directional parameter controlling the vector-based amplitude translation for the direct version and the energy splitting parameter controlling the differential gain for the indirect (decorrelated) version.

[0193] 3GPP Immersive Voice and Audio Services (3GPP Immersive Voice and Audio Services), currently under development, is expected to support new immersive voice and audio services such as mediated reality.

[0194] In some, but not necessarily all, examples, amplitude panning techniques may be used to create or locate sound objects. For example, the known Vector Based Amplitude Panning (VBAP) method may be used to locate sound sources.

[0195] A sound object may be repositioned by mixing a portion of the object's direct form (attenuated and directionally filtered direct sound) with the object's indirect form (e.g., localized directional early reflections and / or diffuse reverberation).

[0196] Figure 14 An example of a method 300 is illustrated, comprising:

[0197] At block 302, content is displayed within a viewport and a symbol representing a sound is displayed within the viewport;

[0198] At block 304 , one or more gestures are detected that place the displayed symbol within the displayed content;

[0199] In box 306, disambiguate between a first placement method performed for the symbol and a second placement method performed for the symbol that is different from the first placement method, the first placement method locking the sound to a position that moves as displayed content moves within the viewport, and the second placement method locking the sound to a position that does not move as the displayed content moves within the viewport, wherein the first placement method includes a gesture for the symbol relative to the viewport that indicates movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

[0200] Figures 15A to 15C An alternative example method of indicating whether a sound source 50 to be added to displayed content 20 is content-locked or viewport-locked is illustrated.

[0201] exist Figure 15AIn FIG. 6 , the user performs an initial gesture 60 to move the symbol 40 representing the sound source 50 to a desired position 61 in the viewport 30 .

[0202] exist Figure 15B In FIG, a prompt 150 is displayed in the viewport 30, inviting the user to input whether the sound source 50 to be added to the displayed content 20 is a content-locked sound or a viewport-locked sound. Figure 15B In the example of FIG. 5 , the prompt is in the form of a selectable icon 152 that is located within the displayed content 20 near the expected location 61 of the sound source 50 .

[0203] exist Figure 15C In , the user inputs 154 indicating whether the sound source 50 is a content-locked sound or a viewport-locked sound. Figure 15C In the example of FIG, user input is made by selecting one of the selectable icons 152 displayed by the prompt 150. The sound source 50 is then defined as content-locked or viewport-locked depending on the input 154.

[0204] When the displayed content 20 is static in the viewport 30, Figures 15A to 15C The method shown in is particularly useful. In this case, it is impossible to disambiguate between the first and second placement methods 101, 102 without further input. When all displayed content 20 is not moving relative to the viewport 30, keeping the symbol 40 static relative to the displayed content 20 within the viewport 30 (placement method 101) and keeping the symbol 40 static relative to the viewport 30 (placement method 102) cannot be disambiguated without further input. This is because following the visual object while the video is static is the same as keeping it stationary. Figures 15A to 15C The method shown in provides further input that can disambiguate. It prompts the user to provide explicit disambiguation by selecting which effect (viewport-locked sound or content-locked sound) the user wants to use for the added sound source 50.

[0205] Figure 16 A method 310 for determining whether to provide a prompt 150 is illustrated.

[0206] At block 312, a determination is made as to whether the displayed content 20 is static for the remaining time duration. If the displayed content 20 is static for the remaining time duration, the method moves to block 312. If the displayed content 20 is not static for the remaining time duration, the method moves to block 316.

[0207] At block 314 , the sound source 50 is added to the desired location 61 (content locked and viewport locked are the same).

[0208] At block 316 , a determination is made as to whether the displayed content 20 is static at the current time.

[0209] If it is determined that the displayed content 20 is static at the current time, then a prompt 150 is generated at block 318 .

[0210] If it is determined that the displayed content 20 is not static at the current time, disambiguation may occur as previously described.

[0211] While method 310 is described as occurring before the process of disambiguation, in other examples it may occur after the process of disambiguation if it fails.

[0212] Figure 17A Illustrated is an example of a controller 400. Such a controller may be used to control the performance of any or all functions of the apparatus 10 and / or perform any of the previously described methods.

[0213] The controller 400 may be implemented as controller circuitry.The controller 400 may be implemented solely in hardware, with certain aspects of software including only firmware, or may be a combination of hardware and software (including firmware).

[0214] like Figure 17A As shown in , the controller 400 can be implemented using instructions that enable hardware functionality, for example, by using executable instructions of a computer program 412 in a general or special purpose processor 402, which executable instructions can be stored on a computer-readable storage medium (disk, memory, etc.) for execution by such processor 402.

[0215] The processor 402 is configured to read from and write to the memory 410. The processor 402 may further include an output interface via which data and / or commands are output by the processor 402 and an input interface via which data and / or commands are input to the processor 402.

[0216] The memory 410 stores a computer program 412 comprising computer program instructions (computer program code) which, when loaded into the processor 402, controls the operation of the apparatus 10. The computer program instructions of the computer program 412 provide the means for enabling the apparatus to perform Figure 1 to Figure 1 5. The processor 402 can load and execute the computer program 412 by reading the memory 410.

[0217] The device 10 thus comprises:

[0218] at least one processor 402; and

[0219] At least one memory 410 including computer program code

[0220] The at least one memory 410 and the computer program code are configured to, together with the at least one processor 402, cause the apparatus 10 to at least perform:

[0221] Display content within the viewport;

[0222] Display symbols representing sounds within the viewport;

[0223] detecting one or more gestures that place the displayed symbol within the displayed content;

[0224] disambiguating between a first placement method for symbolic execution that locks the sound to a position that moves as displayed content moves within a viewport and a second placement method for symbolic execution that is different from the first placement method that locks the sound to a position that does not move as displayed content moves within the viewport,

[0225] The first placement method includes a gesture for the symbol relative to the viewport, the gesture indicating movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

[0226] The apparatus 10 may thus comprise computer program means for

[0227] Display content within the viewport;

[0228] Display symbols representing sounds within the viewport;

[0229] detecting one or more gestures that place the displayed symbol within the displayed content;

[0230] disambiguating between a first placement method for symbolic execution that locks the sound to a position that moves as displayed content moves within a viewport and a second placement method for symbolic execution that is different from the first placement method that locks the sound to a position that does not move as displayed content moves within the viewport,

[0231] The first placement method includes a gesture for the symbol relative to the viewport, the gesture indicating movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

[0232] like Figure 17BAs shown, computer program 412 may arrive at apparatus 10 via any suitable delivery mechanism 420. Delivery mechanism 420 may be, for example, a machine-readable medium, a computer-readable medium, a non-transitory computer-readable storage medium, a computer program product, a memory device, a recording medium (such as a compact disc read-only memory (CD-ROM) or a digital versatile disc (DVD)) or solid-state memory, or an article of manufacture that includes or tangibly embodies computer program 412. The delivery mechanism may be a signal configured to reliably transmit computer program 412. Apparatus 10 may propagate or send computer program 412 as a computer data signal.

[0233] In at least some examples, the computer program instructions are configured to cause the apparatus to at least perform the following operations:

[0234] Display content within the viewport;

[0235] Display symbols representing sounds within the viewport;

[0236] detecting one or more gestures that place the displayed symbol within the displayed content;

[0237] disambiguating between a first placement method for symbolic execution that locks the sound to a position that moves as displayed content moves within a viewport and a second placement method for symbolic execution that is different from the first placement method that locks the sound to a position that does not move as displayed content moves within the viewport,

[0238] The first placement method includes a gesture for the symbol relative to the viewport, the gesture indicating movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

[0239] The computer program instructions may be included in a computer program, a non-transitory computer readable medium, a computer program product, a machine readable medium. In some, but not necessarily all, examples, the computer program instructions may be distributed across more than one computer program.

[0240] Although memory 410 is illustrated as a single component / circuitry, it may be implemented as one or more separate components / circuitry, some or all of which may be integrated / removable and / or may provide permanent / semi-permanent / dynamic / cached storage.

[0241] Although processor 402 is illustrated as a single component / circuitry, it may be implemented as one or more separate components / circuitry, some or all of which may be integrated / removable.Processor 402 may be a single-core or multi-core processor.

[0242] References to "computer-readable storage medium," "computer program product," "tangibly embodied computer program," etc., or "controller," "computer," "processor," etc., should be understood to include not only computers having different architectures (such as single / multi-processor architectures and sequential (Von Neumann) / parallel architectures), but also special-purpose circuits such as field-programmable gate arrays (FPGAs), application-specific circuits (ASICs), signal processing devices, and other processing circuit systems. References to computer programs, instructions, code, etc., should be understood to encompass software for a programmable processor or firmware, such as, for example, the programmable content of a hardware device, whether instructions for a processor or configuration settings for a fixed-function device, gate array, or programmable logic device, etc.

[0243] As used in this application, the term "circuitry" may refer to one or more or all of the following:

[0244] (a) pure hardware circuitry implementation (such as implementation in analog and / or digital circuitry only) and

[0245] (b) a combination of hardware circuitry and software such as (as applicable):

[0246] (i) a combination of analog and / or digital hardware circuits and software / firmware, and

[0247] (ii) any portion of hardware processor(s) (including digital signal processor(s)), software and memory(s) with software that work together to enable a device (such as a mobile phone or server) to perform various functions and

[0248] (c) Hardware circuit(s) and / or processor(s), such as microprocessor(s) or portion(s) of microprocessor(s), require software (e.g., firmware) to operate, but the software may not be present when the software is not required for operation.

[0249] This definition of "circuitry" applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term "circuitry" also covers an implementation of merely hardware circuits or processors and their accompanying software and / or firmware. For example, the term "circuitry" also covers a baseband integrated circuit for a handheld portable device or a similar integrated circuit in a server, cellular network device, or other computing or networking equipment, if applicable to the particular claim element.

[0250] The blocks shown in the figure may represent steps in a method and / or code segments in a computer program 412. The description of a particular order of blocks does not necessarily imply that the blocks have a required or preferred order, and the order and arrangement of the blocks may be changed. In addition, some blocks may be omitted.

[0251] Where a structural feature has been described, it may be replaced by a component that performs one or more of the functions of the structural feature, whether or not that function or functions are described explicitly or implicitly.

[0252] The above example applies the following as the enabler component:

[0253] Automotive systems; telecommunications systems; electronic systems, including consumer electronics; distributed computing systems; media systems for generating or presenting media content, including audio, visual and audiovisual content and mixed, mediated, virtual and / or augmented reality; personal systems, including personal health systems or personal fitness systems; navigation systems; user interfaces also known as human-machine interfaces; networks, including cellular, non-cellular and optical networks; ad hoc networks; the Internet; the Internet of Things; virtualized networks; and related software and services.

[0254] The term "comprising" is used in this document in an inclusive, rather than exclusive, sense. That is, any reference to X including Y indicates that X may include only one Y or may include more than one Y. If the exclusive meaning of "comprising" is intended, the context will be explained by reference to "including only one..." or by the use of "consisting of..."

[0255] In this description, reference is made to various examples. The description of features or functionality associated with an example indicates that those features or functionality are present in that example. Use of the terms "example" or "for example" or "may" or "might" in the text indicates that, whether explicitly stated or not, those features or functionality are present in at least the example being described, whether or not described as an example, and that they may, but need not, be present in some or all other examples. Thus, "example," "for example," "may," or "might" refers to a specific instance within a class of examples. Attributes of an instance may be attributes of only that instance or attributes of a class or a subclass of a class that includes some but not all instances of the class. Thus, it is implicitly disclosed that features described with reference to one example but not with reference to another example may, where possible, be used as part of a working combination in that other example, but need not necessarily be used in that other example.

[0256] Although the embodiments have been described in the preceding paragraphs with reference to various examples, it should be appreciated that modifications to the examples given can be made without departing from the scope of the claims.

[0257] Features described in the preceding description may be used in combinations other than the combinations explicitly described above.

[0258] Although functions have been described with reference to certain features, those functions may be performable by other features whether described or not.

[0259] Although features have been described with reference to certain embodiments, those features may also be present in other embodiments whether described or not.

[0260] As used in this document, the terms "a," "an," and "the" have an inclusive, rather than exclusive, meaning. That is, any reference to X including one or the Y indicates that X may include only one or more than one Y, unless the context clearly indicates otherwise. If the exclusive meaning of "a" or "the" is intended, the context will clearly indicate this. In some cases, "at least one" or "one or more" may be used to emphasize an inclusive meaning, but the absence of these terms should not be construed as inferring an exclusive meaning.

[0261] The presence of a feature (or combination of features) in a claim is a reference to that feature or combination of features itself, as well as to features that achieve substantially the same technical effect (equivalent features). Equivalent features include, for example, features that are variants and achieve substantially the same result in substantially the same way. Equivalent features include, for example, features that perform substantially the same function in substantially the same way to achieve substantially the same result.

[0262] In this specification, reference is made to various examples using adjectives or adjective phrases to describe characteristics of the examples. Such description of a characteristic associated with an example indicates that the characteristic exists exactly as described in some examples and exists substantially as described in other examples.

[0263] The use of the terms "example" or "for example" or "may" or "might" in the text indicates that, whether explicitly stated or not, these features or functions are present in at least the example being described, whether or not described as an example, and that they may, but are not necessarily present in some or all other examples. Thus, "example," "for example," "may," or "might" refers to a particular instance within a class of examples. Properties of an instance may be properties of only that instance or of a class or a subclass of a class that includes some but not all instances of the class. Thus, it is implicitly disclosed that features described with reference to one example and not with reference to another example may, where possible, be used as part of a working combination in that other example, but are not necessarily required to be used in that other example.

[0264] While efforts have been made in the foregoing description to draw attention to those features regarded as important, it will be understood that the applicant may seek protection by way of claims for any patentable feature or combination of features mentioned above and / or shown in the drawings, whether emphasized or not.

Claims

1. An apparatus for communication, comprising means for: Display content within the viewport; displaying a symbol representing a sound source within the viewport; detecting one or more gestures that place the displayed symbol within the displayed content; disambiguating between a first placement method performed on the symbol, the first placement method locking the sound source to a position that moves as displayed content moves within the viewport, and a second placement method performed on the symbol that is different from the first placement method, the second placement method locking the sound source to a position that does not move as displayed content moves within the viewport, The first placement method includes a disambiguating gesture for the symbol relative to the viewport, the disambiguating gesture indicating movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

2. A device according to any preceding claim, wherein The disambiguation gesture of the first placement method includes dragging the symbol relative to the viewport.

3. The device according to claim 2, wherein The first placement method includes a previous gesture that dragged the symbol relative to the viewport and the displayed content prior to the disambiguating gesture.

4. The apparatus according to claim 3, comprising means for: determining a first position in the viewport, the first position defined by the previous gesture dragging an endpoint of the symbol relative to the viewport and the displayed content; and A second position in the viewport is determined, the second position being defined by the termination of the corresponding first placement method or second placement method; wherein a difference between the first position and the second position disambiguates the first placement method and the second placement method.

5. A device according to any preceding claim, wherein The first drop method includes a drop method in which the symbol is dragged to content and then dragged together with the content, and The second placement method includes a placement method in which the symbol is dragged to a position within the viewport and then remains at the position within the viewport.

6. The device according to claim 1, wherein The first placement method and the second placement method include: selection of the symbol and an initial gesture for the symbol relative to the viewport and the displayed content, the initial gesture indicating movement of the symbol relative to the viewport and the displayed content, The first placement method additionally includes: For a gesture of the symbol relative to the viewport, the gesture indicates a movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

7. The device according to claim 1, wherein The first placement method includes: an initial drag movement of the symbol relative to the viewport and the displayed content, and subsequent movement of the symbol relative to the viewport, but without movement of the symbol relative to the displayed content.

8. The device according to claim 1, wherein The first placement method includes: an initial drag movement of the symbol relative to the viewport and the displayed content, and the subsequent movement of the symbol relative to the viewport to a variable position that tracks the displayed content moving within the viewport to which the sound source is to be locked.

9. The apparatus of claim 8, wherein the displayed content that is tracked and to which the sound source is to be locked is a first content of a visual scene.

10. The apparatus of claim 9, wherein the first content is a majority of the visual scene forming a background.

11. The apparatus of claim 9, wherein the first content is a small portion of the visual scene that is a visual object moving within the visual scene.

12. The apparatus of claim 1 , wherein the content displayed within the viewport is a subset of available content, wherein the subset is defined by the viewport, and wherein the viewport is defined by a viewpoint and a field of view, wherein one or both of the displayed content and the viewpoint are time-varying.

13. The device according to claim 1, wherein The first placement method includes a hold gesture that maintains the displayed symbol in a fixed position relative to the content while the viewport moves; and The second placement method includes a hold gesture that maintains the displayed symbol in a fixed position relative to the viewport while the viewport moves.

14. The device according to claim 1, wherein The second placement method performed on the symbol locks the sound source to a position that tracks the viewport rather than to displayed content that is moving relative to the viewport.

15. The device according to claim 1, wherein The second placement method includes: An initial drag movement of the symbol relative to the viewport and the displayed content, and no subsequent movement of the symbol relative to the viewport within a threshold time period.

16. An apparatus according to claim 1, wherein the viewport is a touch display and the symbol is an icon displayed on the touch display, and wherein the one or more gestures to place the displayed symbol within the displayed content are touch-based gestures.

17. The apparatus of claim 16, wherein the icon is provided in a menu, the menu being displayed on the touch display screen.

18. The apparatus of claim 1, wherein the sound source and the content are recorded simultaneously.

19. A method of communication, comprising: Display content within the viewport; displaying a symbol representing a sound source within the viewport; detecting one or more gestures that place the displayed symbol within the displayed content; disambiguating between a first placement method performed on the symbol, the first placement method locking the sound source to a position that moves as displayed content moves within the viewport, and a second placement method performed on the symbol that is different from the first placement method, the second placement method locking the sound source to a position that does not move as displayed content moves within the viewport, The first placement method includes a disambiguating gesture for the symbol relative to the viewport, the disambiguating gesture indicating movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

20. A computer program product comprising program instructions, wherein the program instructions are configured to cause an apparatus to at least perform the following operations: Display content within the viewport; displaying a symbol representing a sound source within the viewport; detecting one or more gestures that place the displayed symbol within the displayed content; and disambiguating between a first placement method performed on the symbol, the first placement method locking the sound source to a position that moves as displayed content moves within the viewport, and a second placement method performed on the symbol that is different from the first placement method, the second placement method locking the sound source to a position that does not move as displayed content moves within the viewport, The first placement method includes a disambiguating gesture for the symbol relative to the viewport, the disambiguating gesture indicating movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

21. A human-machine interface comprising components for: Display content within the viewport; displaying a symbol representing a sound source within the viewport; detecting one or more gestures that place the displayed symbol within the displayed content; disambiguating between a first placement method performed on the symbol, the first placement method locking the sound source to a position that moves as displayed content moves within the viewport, and a second placement method performed on the symbol that is different from the first placement method, the second placement method locking the sound source to a position that does not move as displayed content moves within the viewport, The first placement method includes a disambiguating gesture for the symbol relative to the viewport, the disambiguating gesture indicating movement of the symbol relative to the viewport that disambiguates the first placement method from the second placement method.

Images