Device, method and computer program for rotating visual information for display

Abstract

According to various but not necessarily all embodiments, an apparatus is provided that includes components for the following operations: using at least one captured audio signal to determine the direction of arrival of sound from a sound source relative to a user device; controlling an orientation angle of visual information displayed on a display, wherein the orientation angle of the visual information depends at least in part on the determined direction of arrival of sound from the sound source relative to the user device.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to devices, methods, and computer programs for visual information for a rotating display. Some embodiments relate to devices, methods, and computer programs for visual information for a display in a rotating mobile device. Background Art

[0002] Visual information displayed on a display can be rotated using various methods. Typically, the visual information is rotated so that a user of the display can view the visual information in a particular orientation. Summary of the Invention

[0003] According to various but not necessarily all embodiments, there is provided an apparatus including components for:

[0004] using at least one captured audio signal to determine a direction of arrival of sound from a sound source relative to a user device;

[0005] controlling an orientation angle of visual information displayed on a display, wherein the orientation angle of the visual information depends at least in part on the determined direction of arrival of sound from the sound source relative to the user device.

[0006] In some but not necessarily all examples, the apparatus includes components for controlling a position of the visual information displayed on the display, wherein the position of the visual information displayed on the display depends at least in part on the determined direction of arrival of sound from the sound source relative to the user device.

[0007] In some but not necessarily all examples, the apparatus includes components for determining a change in the direction of arrival of sound from the sound source relative to the user device;

[0008] wherein the components for controlling the orientation angle of the visual information displayed on the display correspondingly change the orientation angle of the visual information in response to the determined change in the direction of arrival of sound from the sound source.

[0009] In some but not necessarily all examples, the apparatus includes components for changing the orientation angle of the visual information to a predefined angle after determining that the determined direction of arrival of sound from the sound source meets a threshold direction of arrival associated with a predefined angle.

[0010] In some but not necessarily all examples, the apparatus includes components for:

[0011] estimating a position of the sound source relative to the user device while the sound source is not producing sound; and

[0012] where the orientation angle of the visual information depends at least in part on the estimated position of the sound source relative to the user device while the sound source is not producing sound.

[0013] In some but not necessarily all examples, the device includes components for changing the scale of the visual information and / or cropping the visual information, where the changing and / or cropping depends at least in part on the orientation angle of the visual information.

[0014] In some but not necessarily all examples, user interface controls associated with the visual information are displayed on the display, where the device includes components for adjusting the position and / or orientation of the user interface controls, where the adjustment depends at least in part on the orientation angle of the visual information.

[0015] In some but not necessarily all examples, the device includes components for:

[0016] determining the distance of the sound source from the user device;

[0017] preventing a change in the orientation angle of the visual information displayed on the display when the determined distance of the sound source is greater than a threshold distance from the user device.

[0018] In some but not necessarily all examples, the orientation angle of the visual information displayed on the display additionally depends on the application settings of an application running on the display, where the application is associated with the visual information.

[0019] In some but not necessarily all examples, the visual information includes one or more images and / or videos captured by an image capture component of the user device;

[0020] where if the user device is rotated while one or more images and / or videos are being captured, the component for controlling the orientation angle of the visual information maintains the orientation angle of the visual information displayed on the display by determining the direction of arrival of the sound from the sound source relative to the rotated user device to counteract a change in the orientation angle of the visual information displayed on the display that does or may occur due to the rotation of the user device.

[0021] In some but not necessarily all examples, the device includes components for:

[0022] identifying the sound source as a previously known sound source; and

[0023] restricting a change in the orientation angle of the visual information, where an unknown sound source is restricted from causing a change in the orientation angle of the visual information.

[0024] In some but not necessarily all examples, the display includes components for enabling one or more gesture controls, where the one or more gesture controls are activated by corresponding gesture inputs of a user, where the device includes components for controlling the adjustment of the one or more gesture controls, where the one or more gesture controls are adjusted when the orientation angle of the visual information is changed such that when a gesture input is performed relative to the changed orientation of the visual information, the gesture input is recognized and the corresponding gesture control is activated; and / or

[0025] where the display includes components for enabling one or more touch controls, where the one or more touch controls are activated by corresponding touch inputs of a user, where the device includes components for controlling the adjustment of the one or more touch controls, where the one or more touch controls are adjusted when the orientation angle of the visual information is changed such that when a touch input is performed relative to the changed orientation of the visual information, the touch input is recognized and the corresponding touch control is activated.

[0026] In some but not necessarily all examples, the device includes components for the following operations:

[0027] Using at least one captured audio signal to determine the direction of arrival of sound from a second sound source relative to the user device;

[0028] Controlling the orientation angle of second visual information displayed on the display, where the visual information and the second visual information are simultaneously displayed on the display, and where the orientation angle of the second visual information depends at least in part on the determined direction of arrival of sound from the second sound source relative to the user device.

[0029] According to various but not necessarily all embodiments, a method is provided, including:

[0030] Using at least one captured audio signal to determine the direction of arrival of sound from a sound source relative to the user device;

[0031] Controlling the orientation angle of visual information displayed on the display, where the orientation angle of the visual information depends at least in part on the determined direction of arrival of sound from the sound source relative to the user device.

[0032] According to various but not necessarily all embodiments, a computer program is provided that, when run on a computer, performs:

[0033] Using at least one captured audio signal to determine the direction of arrival of sound from a sound source relative to the user device;

[0034] Control the orientation angle of visual information displayed on a display, where the orientation angle of the visual information depends at least in part on the determined direction of arrival of sound from a sound source relative to the user device.

[0035] According to various but not necessarily all embodiments, examples as claimed in the appended claims are provided. Brief Description of the Drawings

[0036] Some examples will now be described with reference to the accompanying drawings, in which:

[0037] Figure 1A Examples of the subject matter described herein are shown;

[0038] Figure 1B Examples of the subject matter described herein are shown;

[0039] Figure 2A Examples of the subject matter described herein are shown;

[0040] Figure 2B Examples of the subject matter described herein are shown;

[0041] Figure 3A Examples of the subject matter described herein are shown;

[0042] Figure 3B Examples of the subject matter described herein are shown;

[0043] Figure 4A Examples of the subject matter described herein are shown;

[0044] Figure 4B Examples of the subject matter described herein are shown;

[0045] Figure 4C Examples of the subject matter described herein are shown;

[0046] Figure 5A Examples of the subject matter described herein are shown;

[0047] Figure 5B Examples of the subject matter described herein are shown;

[0048] Figure 5C Examples of the subject matter described herein are shown;

[0049] Figure 6A Examples of the subject matter described herein are shown;

[0050] Figure 6B Examples of the subject matter described herein are shown;

[0051] Figure 7A Examples of the subject matter described herein are shown;

[0052] Figure 7B Shows an example of the subject matter described herein;

[0053] Figure 8A Shows an example of the subject matter described herein;

[0054] Figure 8B Shows an example of the subject matter described herein;

[0055] Figure 9A Shows an example of the subject matter described herein;

[0056] Figure 9B Shows an example of the subject matter described herein;

[0057] Figure 10A Shows an example of the subject matter described herein;

[0058] Figure 10B Shows an example of the subject matter described herein;

[0059] Figure 11A Shows an example of the subject matter described herein;

[0060] Figure 11B Shows an example of the subject matter described herein;

[0061] Figure 12A Shows an example of the subject matter described herein;

[0062] Figure 12B Shows an example of the subject matter described herein;

[0063] Figure 13A Shows an example of the subject matter described herein;

[0064] Figure 13B Shows an example of the subject matter described herein;

[0065] Figure 13C Shows an example of the subject matter described herein;

[0066] Figure 13D Shows an example of the subject matter described herein;

[0067] Figure 14A Shows an example of the subject matter described herein;

[0068] Figure 14B Shows an example of the subject matter described herein;

[0069] Figure 15 Shows an example of the example method described herein;

[0070] Figure 16 illustrates an example computer program described herein;

[0071] Figure 17 illustrates an example of the subject matter described herein. DETAILED DESCRIPTION

[0072] The following description describes an apparatus, method, and computer program that perform the following: use at least one captured audio signal 141 to determine the direction of arrival of sound from a sound source 140 relative to a user device 120; control the orientation angle of visual information 131 displayed on a display 130, where the orientation angle depends at least in part on the determined direction of arrival of sound from the sound source 140 relative to the user device 120.

[0073] The apparatus, method, and computer program provide a technical effect of changing the orientation angle of visual information displayed on a display when other techniques for changing the orientation angle of the visual information are not available or would result in the visual information being incorrectly oriented (which can cause the visual information not to align with the user's perspective of the display). Changing the orientation angle of the visual information causes rotation of the visual information.

[0074] For example, a common problem is that when taking a photo or video using a mobile device while the mobile device camera is facing down towards the ground or up towards the sky, the final result is inadvertently rotated by 90° due to an error in automatic display rotation (e.g., rotation of the display caused by an accelerometer in the mobile device). This also occurs when the mobile device is slightly tilted from facing the ground or the sky. The user may tilt the mobile device to obtain a good fit for the photo or video, or in the case where the mobile device is capturing spatial audio, the user may tilt the device to avoid wind noise when they are outdoors, especially when live monitoring is available for capture.

[0075] This situation is particularly problematic in the case where the video and / or photo is being live streamed, as it may not be possible to correct the inadvertent rotation in post - processing.

[0076] The apparatus, method, and computer program thus solve the technical problem of how to control the orientation angle of visual information displayed on a display. The described apparatus, method, and computer program can be used to control the orientation angle of visual information displayed on a display.

[0077] Example apparatus 100 is illustrated in Figure 1A and Figure 1B .

[0078] Figure 1A Visual information 131 displayed on display 130 is illustrated.

[0079] In Figure 1BIn [the figure], the sound source 140 is illustrated. The sound source 140 generates at least one audio signal 141. The at least one audio signal 141 is captured and used to determine the direction of arrival of the sound from the sound source 140 relative to the user device 120. The apparatus 100 includes components 110 for using the at least one captured audio signal 141 to determine the direction of arrival of the sound from the sound source 140 relative to the user device 120.

[0080] The apparatus 100 includes components 110 for controlling the orientation angle of the visual information 131 displayed on the display 130. In this example, Figure 1B the orientation angle of the visual information 131 in [the figure] has changed compared to Figure 1A the orientation angle in [the figure]. In Figure 1B [the figure], the orientation angle of the visual information depends at least in part on the determined direction of arrival of the sound from the sound source 140 relative to the user device 120.

[0081] The direction of arrival of the sound from the sound source relative to the user device 120 is defined relative to an origin, which is defined as a point associated with the location of the user device 120. For example, the origin can be at the center of the user device 120. The direction of arrival can be defined using three-dimensional bearings (θ, φ), where θ, φ are orthogonal angles, such as the elevation angle φ (which can be represented as the polar angle 90° - φ) and the azimuth angle θ.

[0082] The direction of arrival of the sound can be estimated using, for example, a method based on the time difference of arrival (TDOA) when using two or more microphones or a method based on the horizontal difference when using two or more directional microphones or specific audio sensors constructed for direction detection (such as: Michael Touse et al.: "MEMS directional sound sensor with simultaneous detection of two frequency bands", SENSORS, 2010 IEEE).

[0083] In some examples, the reference plane for the azimuth angle can be the surface of the Earth. In some examples, the elevation angle of the direction of arrival of the sound from the sound source is not used to control the orientation angle of the visual information.

[0084] The orientation angle of the visual information can be defined with respect to a reference orientation of the visual information 131. For example, Figure 1A the orientation of the visual information 131 in [the figure] can be defined as the reference orientation, and the orientation angle of the visual information 131 is thus 90° relative to Figure 1B the reference orientation in [the figure]. Any suitable reference orientation can be defined.

[0085] Figure 2Aand Figure 2B FIG. illustrates another example device 100. In this example, the user device 120 includes the device 100 and the display 130.

[0086] As Figure 2A illustrated, the sound source 140 is at the first position 200 and generates at least one audio signal 141. In Figure 2B it is at the second position 210 and generates at least one audio signal 141. In Figure 2B it, the orientation angle of the visual information 131 has changed compared to its orientation angle in Figure 2A it. The orientation angle depends at least in part on the determined direction of arrival of the sound from the sound source 140 relative to the user device 120.

[0087] In some examples, there is a predefined mapping between the orientation angle at which the visual information 131 is displayed on the display 130 and the determined direction of arrival of the sound from the sound source 140. For example, a particular orientation angle can be associated with a particular determined direction of arrival of the sound. In some examples, a particular orientation angle can be associated with a range of the determined directions of arrival of the sound.

[0088] In this example, the visual information 131 is presented at the orientation angle it has in Figure 2A before the direction of arrival of the sound from the sound source 140 is determined. In this example, Figure 2A the determined direction of arrival of the sound from the sound source 140 in Figure 1A is associated with the orientation angle of the visual information 131 presented in

[0089] Therefore, after the direction of arrival of the sound from the sound source 140 is determined, the orientation angle of the visual information 131 does not change because it has already been associated with the determined direction of arrival of the sound from the sound source 140.

[0089] When the position of the sound source 140 changes to the second position 210 illustrated in Figure 2B the determined direction of arrival of the sound from the sound source 140 changes compared to the determined direction of arrival of the sound in Figure 2A The orientation angle of the visual information 131 changes in Figure 2B compared to its orientation angle in Figure 2A This is because Figure 2B the determined direction of arrival of the sound from the sound source 140 in Figure 2B is associated with the orientation angle of the visual information 131 illustrated in Figure 2A and not with the orientation angle of the visual information 131 illustrated in

[0090] In some examples, the orientation angle of the visual information 131 can be changed regardless of the type of sound generated by the sound source 140. In other examples, the orientation angle of the visual information 131 depends on the device 100 recognizing the sound generated by the sound source 140 as a command to change the orientation angle of the visual information 131 and is changed accordingly. For example, if the sound source 140 is a user, the generated sound can be the user's voice, and in response to recognizing a voice command from the user, the device 100 can change the orientation angle of the visual information 131. In some examples, any change to the orientation angle of the visual information 131 depends on the application associated with the visual information.

[0091] In some examples, the device sets a threshold change in the determined direction of arrival of the sound from the sound source 140 required to trigger a change in the orientation angle of the visual information 131 displayed on the display 130.

[0092] In some examples, the component 110 includes one or more audio sensors, where the one or more audio sensors are configured to capture at least one audio signal 141 for determining the direction of arrival of the sound from the sound source 140 relative to the user device 120. For example, the one or more audio sensors can include one or more microphone arrays. In other examples, the component 110 does not include one or more audio sensors, and one or more audio sensors that are not part of the device 100 provide at least one captured audio signal 141 to the device 100.

[0093] Figure 3A and Figure 3B Another example device 100 is illustrated. In this example, the device includes a component 110 for controlling the position at which the visual information 131 is displayed on the display 130. The position at which the visual information is displayed depends at least in part on the determined direction of arrival of the sound from the sound source 140 relative to the user device 120.

[0094] The visual information 131 is at least at the Figure 3A orientation angle illustrated, since it was at that orientation angle before the direction of arrival of the sound from the sound source 140 was determined, or it can be that the orientation angle of the visual information has been changed to the Figure 3A orientation angle illustrated due to the determined direction of arrival of the sound from the sound source 140.

[0095] As Figure 3A illustrated, there is a space 132 on the display that is closer to the sound source 140 than the position of the visual information 131 in Figure 3A . In Figure 3B , the position of the visual information 131 has changed to occupy at least some of the space in the space 132, as shown by Figure 3BThis means that the visual information 131 is moved closer to the sound source 140. In some examples, the sound source 140 is a user and this provides better visibility of the visual information 131 to the user.

[0096] In some examples, the orientation angle of visual information 131 may be changed before changing to the position of visual information 131. In some examples, the position where visual information 131 is displayed on display 130 may be changed before changing to the orientation angle of visual information 131. In some examples, the position of visual information 131 may be changed simultaneously with the orientation angle of visual information 131.

[0097] In some examples, user interface controls are associated with visual information 131. For example, if visual information 131 is a video, the associated user interface controls may be a play / pause button and / or a volume slider, among others. In some examples, device 100 controls the user interface controls to move with the visual information when the position of visual information 131 is changed.

[0098] Figure 4A , Figure 4B and Figure 4C Another example apparatus 100 is illustrated.

[0099] In this example, the apparatus 100 includes means 110 for determining a change in the direction of arrival of sound from a sound source 140 relative to a user device 120. The means for determining the change in the direction of arrival may be the same means 110 as previously described for determining the direction of arrival. The means for controlling the orientation angle of visual information 131 displayed on the display 130 changes the orientation angle of the visual information 131 correspondingly in response to the determined change in the direction of arrival of the sound from the sound source.

[0100] like Figure 4A , Figure 4B and Figure 4C As shown in FIG. , the location of the sound source 140 is Figure 4A The position 400 in the Figure 4B Position 410 and Figure 4C 4. This causes a change in the determined direction of arrival of the sound from the sound source 140 relative to the user device 120. In response to the determined change in the direction of arrival of the sound from the sound source 140, a corresponding change in the orientation angle of the visual information 131 occurs. By rotating the visual information 131 in this manner, the apparatus 100 aims to maintain the alignment of the orientation angle of the visual information 131 relative to the determined direction of arrival of the sound from the sound source 140.

[0101] In some examples, sound source 140 is a user. By maintaining alignment of the orientation angle of visual information 131 relative to the determined direction of arrival of the sound from sound source 140, this provides the user with the benefit of perceiving visual information 131 as they rotate around display 130 (which in this example is part of user device 120). The user thus observes visual information 131 in substantially the same orientation regardless of their position around display 130.

[0102] This is beneficial in many situations. For example, where the visual information 131 includes a map, it may be beneficial to have the orientation angle of the map maintain its alignment with the determined direction of arrival of the voice from the user, making it easier for the user to read the map while rotating around the display 130. It may be useful in situations where another user is sharing real-time feedback of their location shown on a map forming visual information 131 with the user tracking the location of the other user. By maintaining the alignment of the orientation angle of the map relative to the determined direction of arrival of the voice from the user viewing the map, it is easier for the user to follow the location of the other user, especially when they change direction and / or position on the map.

[0103] In another example, if visual information 131 is text that a user is reading from a display, maintaining alignment of the orientation angle of visual information 131 relative to the determined direction of arrival of sound from the user enables the user to continue to read visual information 131 effortlessly as they change their position around display 130.

[0104] Figure 5A , Figure 5B and Figure 5C Another example device 100 is illustrated. In this example, the device 100 changes the orientation angle of the visual information 131 to the predefined angle 530 after determining that the determined arrival direction of the sound from the sound source 140 satisfies the threshold arrival direction associated with the predefined angle 530. Figure 5A As illustrated in , the visual information 131 is at a first orientation angle. In this example, the sound source 140 is positioned at a first position 500. Figure 5B , the position of the sound source 140 has changed to position 510 and the determined direction of arrival of the sound from the sound source 140 has changed, however the orientation angle of the visual information 131 has not changed. This is because Figure 5B The determined arrival direction of the sound from the sound source 140 in does not satisfy the orientation angle for changing the visual information 131 to Figure 5C The threshold value of the predefined orientation angle 530 is shown in FIG. Figure 5B In the example, the determined arrival direction of the sound from the sound source 140 is still in the same direction as Figure 5Awithin a threshold associated with the orientation angle of the visual information illustrated in the figure.

[0105] In Figure 5C the position of the sound source 140 has been changed to position 520 and the determined direction of arrival of the sound from the sound source 140 has been changed. In Figure 5C the determined direction of arrival of the sound from the sound source 140 satisfies a threshold direction of arrival associated with Figure 5C the predefined orientation angle 530 of the visual information 131 illustrated in the figure. As Figure 5C illustrated in the figure, the orientation angle of the visual information has been changed to the predefined angle 530, which is 90° relative to the orientation angle illustrated in Figure 5A the figure in this example.

[0106] In this example, the user equipment 120 is parallel to the surface of the earth and the reference plane of the azimuth is the surface of the earth. The threshold direction of arrival is that the direction of arrival is not less than 45° of the azimuth in either direction relative to the reference direction of arrival illustrated by the arrow 540, which is perpendicular to the side of the user equipment 120 closest to the arrow 540. In other examples, the threshold direction of arrival associated with the predefined orientation angle 530 is different. In some examples, there may be other predefined orientation angles at which the visual information is rotated. For example, according to Figure 5A , Figure 5B and Figure 5C illustrated in the figure, the visual information 131 may be presented at predefined orientation angles with an increment of 90°. In other examples, the predefined orientation angles may be spaced apart by other increments (such as an increment of 45°).

[0107] Figure 6A and Figure 6B illustrate an example apparatus 100. In this example, the apparatus includes components for: estimating the position 600 of the sound source 140 relative to the user equipment 120 while the sound source 140 is not generating sound; and wherein the orientation angle of the visual information 131 depends at least in part on the estimated position 600 of the sound source 140 relative to the user equipment 120 while the sound source 140 is not generating sound.

[0108] The component 110 may include components for estimating the position of the sound source 140 or it may be provided separately to the apparatus 100.

[0109] The component may include a camera. The device 100 may determine that the object detected by the camera is the sound source 140. Any suitable method may be used to detect the sound source 140 using the camera. For example, if the sound source 140 is the user's voice, then the camera may be used to detect the user's face. The device may determine that the detected face is associated with the user's voice and, thus, the detected face is associated with the sound source 140. The device may determine the direction of the user's face and estimate the distance of the user's face from the user device 120 to estimate the location of the sound source 140.

[0110] In another example, the component for estimating the location 600 includes an Ultra-Wideband (UWB) device. The UWB device is configured to receive a signal from another UWB device indicating the location 600 of the sound source 140. For example, the sound source 140 may have a UWB device associated with it, which enables the location of the sound source 140 relative to the user device 120 to be determined. In some examples, the UWB device associated with the sound source 140 is a mobile device.

[0111] Figure 6A and Figure 6B The examples illustrated in and provide that the location 600 of the sound source 140 can be estimated even while the sound source 140 is not producing sound. This enables the orientation angle of the visual information 131 to be controlled using the location of the sound source and does not rely on the sound source continuously emitting sound or requiring the sound source 140 to emit sound to change the orientation angle of the visual information 131.

[0112] Figure 7A and Figure 7B Illustrates another example device 100. In this example, as illustrated in Figure 7B , the orientation angle of the visual information 131 is changed in response to determining a change in the direction of arrival of the sound from the sound source 140. In this example, the device 100 includes a component 110 for cropping the visual information 131. The cropping depends at least in part on the orientation angle of the visual information 131. In Figure 7A , the displayed visual information 131 includes objects 700, 710, 720. As illustrated in Figure 7B , the orientation angle of the visual information 131 has changed in response to determining a change in the direction of arrival of the sound from the sound source 140. In this example, the position of the visual information 131 on the display has also changed in response to determining a change in the direction of arrival of the sound from the sound source 140. To maintain the scale of the object 700 between Figure 7A and Figure 7B , the visual information 131 has been cropped such that Figure 7AThe objects 710, 720 and other parts of the visual information 131 are no longer displayed. The clipping depends at least in part on the orientation angle of the visual information 131. In this example, it also depends on the position of the visual information 131 in the display.

[0113] The component 110 for clipping the visual information 131 can determine, according to predefined settings, that a particular part of the visual information is of interest and should be maintained after clipping. For example, in Figure 7B , the component for clipping recognizes that the object 700 is a person and should be maintained in the visual information 131 after clipping. This can be beneficial, for example, in the case where the visual information 131 is related to a video call.

[0114] In some examples, the visual information 131 is clipped based on having an optimal fit depending on the type of the visual information 131.

[0115] In some examples, the apparatus 100 includes a component 110 for changing the scale of the visual information 131. The change in the scale of the visual information depends at least in part on the orientation angle of the visual information 131. For example, if it is desired to maintain the entire visual information 131 on the display 130 when the display 130 is rotated, the visual information 131 may need to be reduced in size. The change in the scale of the visual information 131 can also depend on the position of the visual information 131 on the display 130.

[0116] In some examples, the application associated with the visual information 131 can include built-in logic that determines how the visual information 131 should be clipped and / or how the aspect ratio and / or scaling of the visual information 131 should be controlled depending on the orientation angle of the visual information 131 on the display and / or the position of the visual information 131.

[0117] Figure 8A and Figure 8B Illustrates another example apparatus 100. In this example, user interface controls 801, 802 are displayed on the display 130. The user interface controls 801, 802 are associated with the visual information 131 displayed on the display 130. In this example, the apparatus 100 includes a component 110 for adjusting the position and / or orientation of the user interface controls 801, 802, where the adjustment depends at least in part on the orientation angle of the visual information 131. For example, as Figure 8A illustrated, the user interface controls 801, 802 overlay the visual information 131. In Figure 8B , the position of the user interface control 802 has been changed such that it no longer overlays the visual information and the orientation of the user interface control 802 has been adjusted such that it is displayed at the same orientation angle as the visual information 131. In this example, in Figure 8BIn [the example], user interface control 801 is not displayed on the display. This is because user interface control 802 is preferentially displayed in the available space near visual information 131. In other examples, user interface controls 801, 802 may be resized or otherwise positioned such that both user interface controls 801, 802 are displayed in the space near visual information 131.

[0118] In Figure 8B the example, visual information 131 has been cropped, and by rendering user interface control 802 such that it no longer overlays visual information 131, this provides the advantage that the portion of visual information 131 displayed on the display is maximized to achieve a better presentation of visual information to the user while maintaining user interface control 802. In other examples, visual information 131 is not cropped, but component 110 may determine that there is space on display 130 near visual information 131 to place user interface controls 801, 802 in a convenient position for the user while achieving a better presentation of visual information to the user.

[0119] Figure 9A and Figure 9B FIGS. [X] and [Y] illustrate another example apparatus 100. In this example, the apparatus includes component 110 for: determining the distance of sound source 140 from user device 120; preventing a change in the orientation angle of visual information 131 displayed on display 130 if the determined distance of sound source 140 from user device 120 is greater than a threshold distance 900.

[0120] As Figure 9A illustrated in [the figure], sound source 140 is determined to be at a distance 910 from user device 120 that is greater than threshold distance 900. Accordingly, no change in the orientation angle of visual information 131 occurs.

[0121] In Figure 9B the example, the determined distance 920 of sound source 140 from user device 120 is within threshold distance 900, and accordingly the orientation angle of visual information 131 displayed on the display has changed, where the change depends at least in part on the determined direction of arrival of sound from sound source 140 relative to user device 120, and the determined distance of sound source 140 is less than or equal to threshold distance 900.

[0122] To determine the distance of the sound source 140 from the user device 120, the apparatus 100 may include means 110 for determining a direct to ambient measurement from at least one captured audio signal 141. For example, the farther the sound source 140 is from one or more audio sensors used to capture at least one audio signal, the louder the ambient is compared to the direct sound. Direct to ambient measurements are often calculated when capturing spatial audio, and thus in some examples, direct to ambient measurements may be calculated together with determining the direction of arrival of the sound from the sound source 140.

[0123] In some examples, the sound source 140 is the user's voice. In some examples, the sound energy level at common speech frequencies can be used to estimate the distance of the sound source 140 from the user device 120, so that sound signals below a preset or adaptive energy threshold will not be considered relevant, and the device 100 will not react to these to control the orientation angle of the visual information 131.

[0124] In some examples, deep learning may be applied in order to estimate the distance of the sound source 140 from the user device 120. For example, spectral features of at least one captured audio signal 141 are extracted and used in estimating the sound source 140 distance.

[0125] Figure 10A and Figure 10B Another example apparatus 100 is illustrated. In this example, the orientation angle of the visual information 131 displayed on the display 130 is additionally dependent on the application settings of the application running on the display 130. The application is associated with the visual information 131. Figure 10A and Figure 10B In the example illustrated in , the application has an option for rotation lock so that any change in the determined direction of arrival of the sound from the sound source 140 does not change the orientation angle of the visual information 131 associated with the application until the option is deactivated. Figure 10A and Figure 10B As illustrated in FIG. 1 , the position of the sound source has changed from position 1000 to position 1010. The determined arrival direction of the sound from the sound source 140 has been changed to position 1010. Figure 10A and Figure 10B The orientation angle of the visual information 131 has been changed between Figure 10A and Figure 10B Change between.

[0126] The application can have a setting that specifies a threshold change in the angle of the determined direction of arrival of sound from the sound source 140 before any change in the orientation angle that causes the visual information 131. In some examples, the application can specify the direction of arrival of the sound source and / or a range of the direction of arrival of the sound source that will cause a change in the orientation angle of the visual information 131. In some examples, the sound source 140 is the user's voice, and the application has a setting such that the recognized user voice causes a change in the orientation angle of the visual information 131.

[0127] Figure 11A and Figure 11B FIG. illustrates another example device 100. In this example, the visual information 131 includes one or more images and / or videos captured by the image capture component 1110 of the user device 120. In this example, the visual information 131 is displayed within the window 1120. In this example, other visual information 1100 and one or more user interface controls 1101 are also displayed on the display.

[0128] In this example, if the user device 120 is rotated while one or more images and / or videos are being captured, then the component 110 that controls the orientation angle of the visual information 131 maintains the orientation angle of the visual information 131 displayed on the display 130. This is done by determining the direction of arrival of the sound from the sound source 140 relative to the rotating user device 120 to counteract any change in the orientation angle of the visual information 131 that may occur or does occur due to the rotation of the user device 120.

[0129] For example, as Figure 11B illustrated in, the user device 120 has been rotated. The orientation angle of the visual information 131 on the display 130 has been maintained because it is at the same or substantially the same orientation angle on the display 130 as it was in Figure 11A before the rotation. The effect that rotating the user device 120 would typically have on the orientation angle is that one or more images and / or videos displayed on the display 130 are thus counteracted.

[0130] This would be beneficial, for example, in the case where visual information 131 is being streamed to another user using another user device. In this example, the sound source 140 can be the user of the user device 120 that is streaming the visual information 131 to other users. The user (sound source 140) can rotate the user device 120 while they are streaming, for example, they can change their grip while holding the device which causes the device to rotate. Without counteracting the effect that rotating the user device 120 may have on the orientation angle of the streamed visual information, this may cause other users to become dissatisfied with the streamed visual information 131, for example, it may be displayed on the display of the other user device at an orientation angle that is unsatisfactory to the other users.

[0131] In some examples, the apparatus maintains the orientation angle at which the visual information is displayed on the display 130 by treating the position of the sound source 140 as a fixed point in space when the user device 120 rotates. Thus, if a change in the direction of arrival of the sound from the sound source 140 is determined, then the apparatus determines that the user device 120 has been rotated and can determine what rotation has occurred. The position of the sound source 140 can be estimated or determined using any suitable means.

[0132] In some examples, treating the position of the sound source 140 as a fixed point in space is activated by a user indicating means of the user device 120. For example, the apparatus can recognize a voice control from the user or there can be a user interface control on the display 130. In some examples, the application indicating means 100 associated with the visual information 131 treats the position of the sound source 140 as a fixed point in space.

[0133] In some examples, the visual information 131 represents a preview of visual information being streamed to other users on another user device and represents the orientation angle at which the visual information will be displayed on the display of the other user's device. In other examples, the display 130 does not display the visual information 131, and / or does not maintain the orientation angle at which the visual information is displayed on the display 130. In these examples, the apparatus 100 can control the orientation angle of the visual information 131 when the visual information 131 is displayed on the display of the other user's user device to counteract the change in the orientation angle of the visual information displayed on the display due to the rotation of the user device 120.

[0134] In some examples, the orientation angles of other visual information 1100 and one or more user interface controls 1101 on the display 130 do not change when the user device 120 is rotated. This can make it easier for the user to use one or more user interface controls 1101 and view other visual information 1100 in some situations (such as when they are holding the user device 120). In other examples, the orientation angles and / or positions of other visual information 1100 and / or one or more user interface controls may be changed when the user device 120 is rotated and / or when the determined arrival direction of the sound from the sound source 140 changes.

[0135] Figure 12A and Figure 12B Another example device 100 is illustrated. In this example, the device includes components for: identifying the sound source 140 as a previously known sound source; and restricting a change in the orientation angle of the visual information 131. An unknown sound source, such as the sound source 1200 that generates at least one audio signal 1210, is restricted from causing a change in the orientation angle of the visual information 131.

[0136] As Figure 12B illustrated, the orientation angle of the visual information 131 has changed from its orientation angle in Figure 12A in response to determining the arrival direction of the sound from the sound source 140. The orientation angle of the visual information 131 has not been changed by the arrival direction of the sound from the sound source 1200.

[0137] The component 110 may use stored information related to the known sound source or be able to access information related to the known sound source.

[0138] Figure 12A and Figure 12B The example illustrated in

[0139] Figure 13A and Figure 13B Another example device 100 is illustrated. In this example, the display includes components for enabling one or more gesture controls, where the one or more gesture controls are activated by corresponding gesture inputs 1310 of the user, where the device includes a component 110 for controlling an adjustment to the one or more gesture controls, and where the one or more gesture controls are adjusted when the orientation angle of the visual information 131 is changed such that when the gesture input 1310 is performed relative to the changed orientation angle of the visual information 131, the gesture input 1310 is recognized and the corresponding gesture control is activated.

[0140] In Figure 13A and Figure 13B , region 1300 represents the area of the display where specific gesture controls can be detected. For example, the gesture control can be a control that enables changing the volume of the audio associated with visual information 131. When the visual information 131 is presented at the Figure 13A orientation angle, the gesture control is activated by a gesture input indicated by arrow 1310. In some examples, the gesture input is a proximity input, where the user can place their finger above region 1300 and move their finger as indicated by arrow 1310 to change the volume of the audio associated with the visual information.

[0141] In Figure 13B , the orientation angle of the visual information 131 has changed in response to determining the direction of arrival of sound from sound source 140. Region 1300 has rotated and the gesture input recognized as activating the gesture control is illustrated by arrow 1320. The device thus implements an adjustment of the gesture control in response to a change in the orientation angle of the visual information 131.

[0142] In other examples, the gesture input is the movement of the user's hand and / or arm detected by, for example, the motion detection component of device 100. When the orientation angle of the visual information 131 is presented at the Figure 13A orientation angle, the gesture control is activated by the movement of the user's hand and / or arm detected by the motion detection component in the direction indicated by arrow 1301. This can be, for example, a swiping motion of the user's hand and / or arm.

[0143] When the orientation angle of the visual information 131 is presented at the Figure 13B orientation angle, the gesture control is activated by the movement of the user's hand and / or arm detected by the motion detection component in the direction indicated by arrow 1320. The device thus implements an adjustment of the gesture control in the case where the gesture input is the movement of the user's hand and / or in response to a change in the orientation angle of the visual information 131.

[0144] In some examples, the gesture input is a sequence of movements of the user's hand and / or arm detected by the motion detection component.

[0145] The gesture input can be performed at any position within the field of view of the motion detection component and is thus not limited to a specific area of the display, as in other examples.

[0146] The motion detection component can include, for example, a camera and / or a depth sensor. Component 110 can have built-in logic for recognizing the detected movement as a gesture input and for adjusting what movements relative to the motion detection component are detected as gesture inputs.

[0147] One or more gesture controls may be related to gesture controls that control one or more features of an application associated with visual information 131. For example, as Figure 13A and Figure 13B illustrated, a volume control may be associated with an application. In some examples, one or more gesture controls may relate to one or more system menus 1330 associated with the display 130. For example, in the case where the user device includes a display, the system menu 1330 of the user device may be activated by a gesture control, and the gesture control may be adjusted by rotation of the visual information. As Figure 13C and Figure 13D illustrated, the arrow 1340 represents a gesture input that can activate and deactivate the system menu 1330. For example, applying a gesture input in one direction of the arrow 1340 activates the system menu, and the other direction of the arrow 1340 deactivates the system menu. As Figure 13D illustrated, the determined direction of arrival of the sound from the sound source 140 has changed, and the gesture control has been rotated such that the arrow 1350 represents the gesture input recognized to activate the gesture control. As Figure 13D illustrated, the orientation angle at which the system menu 1330 is displayed on the display 130 depends on the determined orientation angle of the sound from the sound source 140. The system menu 1330 may overlay the visual information when activated.

[0148] Figure 13A 、 Figure 13B 、 Figure 13C and Figure 13D illustrated examples illustrate examples of one or more gesture controls and corresponding gesture inputs. In some examples, the display 130 may include components for enabling one or more touch controls, where the one or more touch controls are activated by corresponding touch inputs, where the device 100 includes components for controlling the adjustment of the one or more touch controls, where the one or more touch controls are adjusted when the orientation angle of the visual information 131 is changed such that when a touch input is performed relative to the changed orientation of the visual information, the touch input is recognized and the corresponding touch control is activated.

[0149] In these examples, the area 1300 represents the area of the display where a particular touch control can be detected. The arrows 1310, 1320, 1340, 1350 represent touch inputs that activate the corresponding touch controls. One or more touch controls may operate in conjunction with gesture controls. In some examples, the touch controls and gesture controls relate to the same control. For example, Figure 13A and Figure 13B illustrated volume controls may be controlled by both touch controls and gesture controls. In some examples, one or more touch controls are provided that provide different controls from one or more gesture controls.

[0150] Figure 14A and Figure 14B illustrates another example device 100. In this example, the device includes components for the following operations: using at least one captured audio signal 1401 to determine the direction of arrival of sound from a second sound source 1400 relative to the user device 120; controlling the orientation angle of second visual information 1410 displayed on a display, where visual information 131 and second visual information 1410 are simultaneously displayed on the display 130, and where the orientation angle of the second visual information 1410 depends at least in part on the determined direction of arrival of sound from the second sound source 1400 relative to the user device 120.

[0151] In Figure 14A and Figure 14B 's example, visual information 131 and second visual information 1410 are initially the same piece of visual information 131 as illustrated in Figure 14A . In response to determining that the sound source 140 and the second sound source 1400 are detected using the corresponding directions of arrival of sound, the visual information is presented as two separate pieces of visual information, such as the visual information 131 and the second visual information 1410 illustrated in Figure 14B . In some examples, multiple copies of the same visual information 131 can be displayed on the display, and the determined directions of arrival of sound from multiple sound sources can each individually control the orientation angle and / or position of a copy associated with that sound source 140.

[0152] In other examples, several different pieces of visual information can be displayed on the display 130, which can be associated with different applications. This enables multiple users to use the same display 130 to view different visual information. In some examples, there are more than two detected sound sources, and each sound source can interact with a different piece of visual information. In the case where the sound source is the user's voice, each piece of visual information can be associated with a specific user by assigning the visual information to a specific voice. The device can control the different pieces of visual information so that they do not overlap with each other when the sound sources change their positions and change the orientation angle and / or position of the visual information.

[0153] Gestures and / or touch controls associated with each piece or copy of visual information can be provided and can be adjusted accordingly as the orientation angle and / or position of the visual information changes.

[0154] Figure 15FIG. illustrates an example method 1500. The method includes: determining, using at least one captured audio signal 141 as illustrated in block 1510, the direction of arrival of sound from a sound source 140 relative to a user device 120. The method further includes controlling, as illustrated in block 1520, the orientation angle of visual information 131 displayed on a display 130. The orientation angle of the visual information 131 depends at least in part on the determined direction of arrival of sound from the sound source 140 relative to the user device 120.

[0155] Figure 16 FIG. illustrates an example computer program 1600 that, when run on a computer, performs: determining, using at least one captured audio signal 141 as illustrated in block 1610, the direction of arrival of sound from a sound source 140 relative to a user device 120; controlling, as illustrated in block 1620, the orientation angle of visual information 131 displayed on a display 130. The orientation angle of the visual information 131 depends at least in part on the determined direction of arrival of sound from the sound source 140 relative to the user device 120.

[0156] The examples illustrated and described above in Figures 1A to 14B may be provided as additional blocks in method 1500 and / or computer program 1600.

[0157] The examples illustrated and described above in Figures 1A to 14B may all be implemented in apparatus 100. In some examples, a combination of one or more examples may be selected by user preference. In cases where the examples are mutually exclusive, they may be provided as alternative settings of applications or settings of apparatus 100 and / or user device 120 and / or display 130.

[0158] In Figures 2A to 14B the example illustrated, the user device 120 includes apparatus 100 and display 130. Figures 2A to 14B The example illustrated in

[0159] may also be implemented in cases where apparatus 100, user device 120, and display 130 are separate or in cases where any two of them are combined. Communication between apparatus 100, user device 120, and display 130 may be provided by a physical interface, a wireless interface, or a combination thereof.

[0160] In some example apparatus, methods, and computer programs, the examples illustrated and described above in Figures 1A to 16The examples illustrated herein can be used when determining that the plane of the display 130 is parallel to the surface of the Earth or within a threshold of being tilted from the surface of the Earth. This can provide the advantage that the orientation angle of the visual information 131 displayed on the display can be changed while other methods of changing the orientation angle are unavailable or unreliable. Other methods can be used when the display 130 is tilted beyond the threshold. Other methods include, for example, using an accelerometer associated with the display 130.

[0161] In some examples, a device includes an ultra-wideband (UWB) device that can be used to determine the direction of a sound source from a user device. In these examples, the sound source 140 is associated with the UWB device, which communicates with the UWB device of the device. This can be used in combination with or instead of determining the direction of arrival of the sound from the sound source 140.

[0162] In the examples illustrated above, the user device can be a mobile device. The user device can be a tablet device. The display can be the display of the mobile device. The display can be the display of the tablet device. The display can be a large screen, which can be integrated or mounted on a surface.

[0163] Figure 17 An example component 110 used in the examples illustrated above is shown. In Figure 17 the component 110 includes a controller 111. The implementation of the controller 111 can be as controller circuitry. The controller 111 can be implemented only in hardware, with certain aspects in software (including only firmware) or can be a combination of hardware and software (including firmware).

[0164] As Figure 17 shown, the controller 111 can be implemented using instructions that implement hardware functions, such as by using executable instructions of a computer program 114 that can be stored on a computer-readable storage medium (disk, memory, etc.) to be executed by such a processor 112 in a general-purpose or special-purpose processor 112.

[0165] The processor 112 is configured to read from and write to the memory 113. The processor 112 can also include an output interface through which data and / or commands are output by the processor 112 and an input interface through which data and / or commands are input to the processor 112.

[0166] The memory 113 stores a computer program 114 that includes computer program instructions (computer program code) that control the operation of the device 100 when loaded into the processor 112. The computer program instructions of the computer program 114 provide for enabling the device to perform Figure 15 and Figure 16The logic and routines of the method illustrated in the figure. The processor 112 can load and execute the computer program 114 by reading the memory 113.

[0167] Thus, in some examples, the apparatus 100 thus includes:

[0168] At least one processor 112; and

[0169] At least one memory 113, including computer program code,

[0170] The at least one memory 113 and the computer program code are configured to, together with the at least one processor 112, cause the apparatus 100 to at least perform:

[0171] Determine the direction of arrival of the sound from the sound source 140 relative to the user equipment 120 using at least one captured audio signal 141;

[0172] Control the orientation angle of the visual information 131 displayed on the display 130, wherein the orientation angle of the visual information 131 depends at least in part on the determined direction of arrival of the sound from the sound source 140 relative to the user equipment 120.

[0173] As Figure 17 Illustrated in, the computer program 114 can reach the apparatus 100 via any suitable delivery mechanism 1700. The delivery mechanism 1700 can be, for example, a machine-readable medium, a computer-readable medium, a non-transitory computer-readable storage medium, a computer program product, a memory device, a recording medium such as a compact disc read-only memory (CD-ROM) or a digital versatile disc (DVD), or a solid-state memory, an article of manufacture including or tangibly embodying the computer program 114. The delivery medium can be a signal configured to reliably transmit the computer program 114. The apparatus 100 can propagate or transmit the computer program 114 as a computer data signal.

[0174] Computer program instructions can be provided to cause the apparatus 100 to at least perform the following operations or to at least perform the following operations:

[0175] Cause the determination of the direction of arrival of the sound from the sound source 140 relative to the user equipment 120 using at least one captured audio signal 141;

[0176] Cause the control of the orientation angle of the visual information 131 displayed on the display 130, wherein the orientation angle of the visual information 131 depends at least in part on the determined direction of arrival of the sound from the sound source 140 relative to the user equipment 120.

[0177] Computer program instructions may be included in a computer program, a non-transitory computer-readable medium, a computer program product, or a machine-readable medium. In some but not necessarily all examples, the computer program instructions may be distributed over more than one computer program.

[0178] Although the memory 113 is illustrated as a single component / circuit system, it may be implemented as one or more separate components / circuit systems, some or all of which may be integrated / removable and / or may provide permanent / semi-permanent / dynamic / cache storage devices.

[0179] Although the processor 112 is illustrated as a single component / circuit system, it may be implemented as one or more separate components / circuits, some or all of which may be integrated / removable. The processor 112 may be a single-core or multi-core processor.

[0180] References to "computer-readable storage medium", "computer program product", "tangibly implemented computer program", etc. or "controller", "computer", "processor", etc. should be understood to include not only computers having different architectures such as single / multiple processor architectures and sequential (von Neumann) / parallel architectures but also specialized circuits such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), signal processing devices, and other processing circuit systems. References to computer programs, instructions, code, etc. should be understood to include software or firmware for programmable processors, such as, for example, programmable vision information for a hardware device, whether instructions for a processor or configuration settings for a fixed function device, gate array, or programmable logic device, etc.

[0181] As used in this application, the term "circuit system" refers to one or more or all of the following:

[0182] (a) Only hardware circuit system implementations (such as implementations in only analog and / or digital circuit systems)

[0183] (b) Combinations of hardware circuits and software, such as (if applicable):

[0184] (i) Combinations of (multiple) analog and / or digital hardware circuits with software / firmware and

[0185] (ii) Any portion of (multiple) hardware processors (including (multiple) digital signal processors), software, and (multiple) memories that work together to cause a device such as a mobile phone or a server to perform various functions, and

[0186] (c) One or more hardware circuits and / or one or more processors, such as one or more microprocessors or a portion of one or more microprocessors, which require software (e.g., firmware) for operation, but the software may be absent when it is not required for operation.

[0187] This definition of circuitry applies throughout this application, including all uses of the term in any claims. As another example, as used throughout this application, the term circuitry also encompasses implementations of only hardware circuits or processors and their accompanying software and / or firmware. For example and if applicable to a particular claim element, the term circuitry also encompasses a baseband integrated circuit for a mobile device or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0188] Figure 15 and Figure 16 The blocks illustrated in and may represent steps in a method and / or code segments in a computer program 114. The illustration of a particular order of the blocks does not necessarily imply a requirement or a preferred order for the blocks, and the order and arrangement of the blocks may be changed. Additionally, some blocks may be omitted.

[0189] In the example described above, component 110 may use any suitable method to determine the direction of arrival of sound from a sound source relative to user device 120 using at least one captured audio signal 141. For example, processor 112 may determine the direction of arrival of sound from the sound source. Component 110 may be arranged to receive at least one captured audio signal from a plurality of audio sensors. One or more audio sensors may be spatially distributed such that at least one captured audio signal represents spatial sound.

[0190] In some examples, the plurality of audio sensors includes a microphone array. The microphones within the array may be arranged within the array such that they enable spatial sound to be recorded.

[0191] Processor 112 within component 110 may be configured to obtain spatial metadata using at least one captured audio signal 141. The spatial metadata corresponds to the spatial sound represented by the at least one audio signal. The spatial metadata includes information related to the spatial attributes of the sound space recorded by the microphones. The spatial metadata may be determined in a frequency band. The spatial metadata may include information such as direction of arrival, distance, and direct-to-total energy ratio for each frequency band in the signal.

[0192] To determine the direction of arrival of an audio signal, audio sensors can be spaced apart such that the direction of arrival of the audio signal can be determined from the phase and amplitude differences measured by one or more audio sensors. Information related to the relative positions of the audio sensors with respect to each other and with respect to the user device can be known to the processor 112 (e.g., it is stored in the memory 113) or can be provided to the controller 111. The processor 112 defines the origin from which the direction of arrival is determined. For example, the origin is defined as the location of the user device 120.

[0193] Component 110 can use any suitable method to determine whether the sound source 140 or any other sound source is present from at least one captured audio signal. For example, to determine whether a sound source is present, the processor 112 can use the methods and information stored in the memory 113. This will depend on the type of sound source to be detected. In an example where the sound source can be any sound, the processor 112 is configured to identify the sound from at least one captured audio signal by the frequency pattern associated with the sound. In some examples, such as Figure 12A and Figure 12B as illustrated, the processor 112 can determine that the sound source is the previously known sound source 140. In the case where the sound source is the user's voice, the processor 112 can use the information and methods stored in the memory 113 (such as the voice patterns of known users) to determine that the sound source is a known sound source.

[0194] In some examples, component 110 changes the orientation angle of the visual information or any other change to the visual information described above in response to a specified command from the user. For example, the processor 112 can use at least one captured audio signal to determine whether the user has spoken a command.

[0195] In other examples, the sound source 140 is not the user's voice. In these examples, the sound source can be any sound source that produces a sound that component 110 can identify. For example, the sound source can produce an audio signal having a specific frequency pattern, amplitude pattern, a combination of both, or any other audio attribute that can be used to identify the sound source.

[0196] In some examples, to change the visual information 131 displayed on the display 130, the controller 111 is configured to control the display. For example, the device 100 and the display 130 are part of the same device. In other examples, in the case where the device 100 and the display 130 are in separate devices, the controller 111 may provide instructions to the controller of the device that controls the display. Similarly, in examples where the device changes user interface controls and / or gesture controls and / or touch controls, in some examples, the device 100 and the display 130 are part of the same device, and the controller 111 is configured to change the controls and inputs it recognizes. In other examples where the device 100 and the display 130 are not part of the same device, the controller 111 is configured to provide instructions to the controller of the display 130 to change the controls and inputs it is configured to recognize.

[0197] Although structural features have been described, they may be replaced by components for performing one or more of the functions of the structural features, whether the function or functions are explicitly or implicitly described.

[0198] The recording of data may include only temporary recording, or it may include permanent recording, or it may include both temporary and permanent recording, where temporary recording implies a temporary recording of data. This may occur, for example, during sensing or image capture, at dynamic memory, at buffers such as circular buffers, registers, caches, or the like. Permanent recording implies that the data is in the form of an addressable data structure that is retrievable from an addressable memory space and can thus be stored and retrieved until deleted or rewritten, although long-term storage may or may not occur. The use of the term "capture" in relation to an image refers to the temporary recording of the data of the image. The use of the term "store" in relation to an image refers to the permanent recording of the data of the image.

[0199] The example applications described above are for implementing the following components: automotive systems; communication systems; electronic systems including consumer electronics; distributed computing systems; media systems for generating or presenting media visual information including audio, visual, and audiovisual information as well as mixed, mediated, virtual, and / or augmented reality; personal systems including personal health systems or personal fitness systems; navigation systems; user interfaces also referred to as human-machine interfaces; networks including cellular, non-cellular, and optical networks; ad-hoc networks; the Internet; the Internet of Things; virtualized networks; and associated software and services.

[0200] The term "comprising" is used in this document in an inclusive rather than exclusive sense. That is, any reference to X comprising Y indicates that X may comprise only one Y or may comprise more than one Y. If it is intended to use "comprising" in an exclusive sense, then it will be clearly stated in the context by reference to "comprising only one.." or by using "consisting of".

[0201] In this specification, various examples have been referred to. The description of a feature or function in relation to an example indicates that those features or functions exist in that example. Whether or not explicitly stated, the use of the terms "example" or "for example" or "able to" or "may" in this text indicates that such features or functions exist in at least the described example, whether or not described as an example, and they may but need not exist in some or all other examples. Thus, "example", "for example", "able to" or "may" refer to a particular instance within a class of examples. The attributes of that instance may be attributes of only that instance or attributes of the class or attributes of a subclass of the class that includes some but not all of the instances in the class. Thus, features described with reference to one example but not to another example are implicitly disclosed and may, where possible, be used in that other example as part of a working combination but need not necessarily be used in that other example.

[0202] Although examples have been described in the previous paragraphs with reference to various examples, it should be recognized that modifications to the given examples can be made without departing from the scope of the claims.

[0203] The features described in the foregoing description can be used in combinations other than those explicitly described above.

[0204] Although functions have been described with reference to certain features, those functions can be performed by other features, whether or not described.

[0205] Although features have been described with reference to certain examples, those features may also exist in other examples, whether or not described.

[0206] The term "a" or "the" is used in this document in an inclusive rather than exclusive sense. That is, unless the context clearly gives a contrary indication, any reference to X comprising a / the Y indicates that X may comprise only one Y or may comprise more than one Y. If it is intended to use "a" or "the" in an exclusive sense, then it will be clearly stated in the context. In some cases, the use of "at least one" or "one or more" may be used to emphasize the inclusive sense, but the absence of these terms should not be taken to imply any exclusive sense.

[0207] The presence of a feature (or combination of features) in a claim is a reference to that feature or combination of features itself and also to features (equivalent features) that achieve substantially the same technical effect. Equivalent features include, for example, features that are variants and achieve substantially the same result in substantially the same way. Equivalent features include, for example, features that perform substantially the same function in substantially the same way to achieve substantially the same result.

[0208] In this specification, various examples have been referred to in which an adjective or adjective phrase is used to describe a characteristic of an example. Such a description of a characteristic associated with an example indicates that the characteristic is present in some of the examples precisely described and in other examples substantially described.

[0209] Although the foregoing specification has been devoted to focusing on those features considered to be important, it should be understood that the applicant may seek protection via the claims in respect of any patentable feature or combination of features mentioned hereinbefore and / or shown in the drawings (whether or not emphasis has been placed thereon).

Claims

1. A device comprising components for the following operations: Using at least one captured audio signal to determine the direction of arrival of sound from a sound source relative to the user device and the distance of the sound source from the user device; In response to the determined distance of the sound source being less than a threshold distance from the user device, controlling the orientation angle of visual information displayed on a display, wherein the orientation angle of the visual information depends at least in part on the determined direction of arrival of sound from the sound source relative to the user device; And In response to the determined distance of the sound source being greater than a threshold distance from the user device, maintaining the orientation angle of the visual information displayed on the display from changing by preventing the change of the orientation angle of the visual information displayed on the display based on the determined direction of arrival of sound from the sound source relative to the user device.

2. The device according to claim 1, comprising components for controlling the position at which the visual information is displayed on the display, wherein the position of the visual information displayed on the display depends at least in part on the determined direction of arrival of sound from the sound source relative to the user device.

3. The device according to any one of the preceding claims, wherein the device comprises components for determining a change in the direction of arrival of sound from the sound source relative to the user device; Wherein the components for controlling the orientation angle of the visual information displayed on the display correspondingly change the orientation angle of the visual information in response to the determined change in the direction of arrival of sound from the sound source.

4. The device according to any one of claims 1 to 2, wherein the device comprises components for changing the orientation angle of the visual information to a predefined angle after determining that the determined direction of arrival of sound from the sound source satisfies a threshold direction of arrival associated with a predefined angle.

5. The device according to any one of claims 1 to 2, comprising components for the following operations: Estimating the position of the sound source relative to the user device while the sound source is not producing sound; and Wherein the orientation angle of the visual information depends at least in part on the estimated position of the sound source relative to the user device while the sound source is not producing sound.

6. The device according to any one of claims 1 to 2, comprising components for changing the scale of the visual information and / or cropping the visual information, wherein the change and / or the cropping depend at least in part on the orientation angle of the visual information.

7. The device according to any one of claims 1 to 2, wherein user interface controls associated with the visual information are displayed on the display, wherein the device comprises components for adjusting the position and / or orientation of the user interface controls, wherein the adjustment depends at least in part on the orientation angle of the visual information.

8. The apparatus according to any one of claims 1 to 2, wherein the orientation angle of the visual information displayed on the display additionally depends on an application setting of an application running on the display, wherein the application is associated with the visual information.

9. The apparatus according to any one of claims 1 to 2, wherein the visual information includes one or more images and / or videos captured by an image capture component of the user equipment; wherein if the user equipment is rotated while the one or more images and / or videos are being captured, the component for controlling the orientation angle of the visual information maintains the orientation angle of the visual information displayed on the display by determining the arrival direction of the sound from the sound source relative to the rotated user equipment to offset a change in the orientation angle of the visual information displayed on the display that does occur or may occur due to the rotation of the user equipment.

10. The apparatus according to any one of claims 1 to 2, comprising components for: identifying the sound source as a previously known sound source; and restricting a change in the orientation angle of the visual information, wherein an unknown sound source is restricted from causing a change in the orientation angle of the visual information.

11. The apparatus according to any one of claims 1 to 2, wherein the display includes components for enabling one or more gesture controls, wherein the one or more gesture controls are activated by a corresponding gesture input of a user, wherein the apparatus includes components for controlling an adjustment of the one or more gesture controls, wherein the one or more gesture controls are adjusted when the orientation angle of the visual information is changed such that when a gesture input is performed relative to the changed orientation of the visual information, the gesture input is recognized and the corresponding gesture control is activated; and / or wherein the display includes components for enabling one or more touch controls, wherein the one or more touch controls are activated by a corresponding touch input of a user, wherein the apparatus includes components for controlling an adjustment of the one or more touch controls, wherein the one or more touch controls are adjusted when the orientation angle of the visual information is changed such that when a touch input is performed relative to the changed orientation of the visual information, the touch input is recognized and the corresponding touch control is activated.

12. The apparatus according to any one of claims 1 to 2, wherein the apparatus includes components for: using at least one captured audio signal to determine the arrival direction of the sound from a second sound source relative to the user equipment; controlling the orientation angle of second visual information displayed on the display, wherein the visual information and the second visual information are simultaneously displayed on the display, wherein the orientation angle of the second visual information depends at least in part on the determined arrival direction of the sound from the second sound source relative to the user equipment.

13. A method for rotating visual information for display, comprising: Determine the direction of arrival of sound from a sound source relative to a user device and the distance of the sound source from the user device using at least one captured audio signal; In response to the determined distance of the sound source being less than a threshold distance from the user device, control the orientation angle of visual information displayed on a display, wherein the orientation angle of the visual information depends at least in part on the determined direction of arrival of sound from the sound source relative to the user device; And In response to the determined distance of the sound source being greater than a threshold distance from the user device, maintain the orientation angle of the visual information displayed on the display from changing by preventing a change in the orientation angle of the visual information displayed on the display based on the determined direction of arrival of sound from the sound source relative to the user device.

14. A computer program that, when run on a computer, performs: Determine the direction of arrival of sound from a sound source relative to a user device and the distance of the sound source from the user device using at least one captured audio signal; In response to the determined distance of the sound source being less than a threshold distance from the user device, control the orientation angle of visual information displayed on a display, wherein the orientation angle of the visual information depends at least in part on the determined direction of arrival of sound from the sound source relative to the user device; And In response to the determined distance of the sound source being greater than a threshold distance from the user device, maintain the orientation angle of the visual information displayed on the display from changing by preventing a change in the orientation angle of the visual information displayed on the display based on the determined direction of arrival of sound from the sound source relative to the user device.

Images