Systems and camera devices for capturing images

Abstract

A system for adjusting a posture of a camera relative to a subject in a scene is provided. The system includes: a camera operable to capture an image of a scene; a recognition unit configured to recognize an object of interest in the image of the scene; a posture processor configured to obtain a posture of the object of interest in the scene relative to the camera; and a scene analyzer operable to determine a scene quality associated with an image captured by the camera based on at least one of the obtained posture of the object of interest and the image captured by the camera. A controller is configured to cause the posture of the camera to be adjusted based on a determination that the scene quality of an image captured at a current posture is less than a threshold. A corresponding device is also provided.

Description

Technical Field

[0001] The present disclosure relates to systems and camera devices for capturing images. Background Art

[0002] The accessibility of photography and videography has increased tremendously in recent times. This has been spurred, at least in part, by the proliferation of portable camera devices available on the market, ranging from smartphone devices, wearable cameras, professional cameras, to drones with cameras incorporated into them.

[0003] While the resolution of camera equipment continues to improve, the quality of images captured by such cameras depends heavily on the skill of the operator. For example, characteristics such as composition, subject occlusion, lighting, depth of field, etc. often depend on how the operator configures the camera relative to the subject in the scene. For scenes containing complex terrain or multiple dynamic subjects, positioning the camera to obtain the best image is not always simple.

[0004] Some camera devices (such as Clip cameras) are trained to automatically detect events of interest occurring in a scene and capture images of those events. While this is certainly useful, the quality of images captured in this way still depends heavily on where the operator points the camera. Additionally, these devices tend to be stationary, so coverage of dynamic scenes is limited.

[0005] Camera devices that track objects of interest are well known in the art. For example, so-called "follow me" drones are capable of tracking a user based on detection of the user's location (e.g., using GPS signals or computer vision associated with the user). However, these drones typically maintain a predetermined relative position to the user and do not respond to any detected occlusion of the subject or degradation of the aesthetic quality of the captured image.

[0006] Some video games provide a virtual camera for capturing images of a virtual scene, such as an in-game environment. The pose of this virtual camera is usually controlled by the player, so the quality of the captured images depends on the player's control of the virtual camera. In some games, the pose of the camera can be controlled using pre-configured script paths, for example, to maintain a certain distance from a point on the avatar's body, like the virtual version of the drone described above.

[0007] However, in general, positioning a virtual camera in this way does not always result in the best image of the captured virtual scene.

[0008] The present invention seeks to solve or at least alleviate these problems. Summary of the invention

[0009] According to a first aspect disclosed herein, a system according to claim 1 is provided.

[0010] According to a second aspect disclosed herein, there is provided an apparatus according to claim 15. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] To aid in understanding the present disclosure and to show how embodiments may be implemented, reference is made to the accompanying drawings by way of example, in which:

[0012] Figure 1 An example of a scene including two subjects and cameras having different postures relative to the subjects is shown; and

[0013] Figure 2 An example of a system for capturing an image of a scene is schematically shown. DETAILED DESCRIPTION

[0014] A system and camera device for capturing images are disclosed. In the following description, some specific details are shown to provide a thorough understanding of embodiments of the present invention. However, it is apparent to those skilled in the art that these specific details need not be utilized to implement the present invention. Instead, specific details known to those skilled in the art have been omitted for clarity where appropriate.

[0015] For various reasons, a camera operator may wish to capture an image of a scene. In some examples, the scene may correspond to a real physical environment, and the user may wish to capture still images or video images of the environment. Typically, this is to capture images of one or more subjects in the scene. The subjects may correspond to, for example, people in the scene, the camera operator himself (e.g., in the case of a selfie), specific objects in the scene, etc.

[0016] In some examples, the scene may be a virtual scene, such as a virtual environment rendered as part of a video game, and the user may wish to capture still or video images of events in the game using a virtual camera. For example, a player may wish to capture an image of his or her avatar in a video game.

[0017] Figure 1 An example of a scene 100 including two subjects 102, 104 is shown. Figure 1, the subject corresponds to a person, but could correspond to any item of interest. In this case, the scene includes another object (tree 106). The camera is shown at a first position and orientation (i.e., pose) 110A relative to the subject in the scene. An image captured by the camera at the pose is displayed as a first image 110A. The camera is also shown at a second pose 110B relative to the subject in the scene, and a corresponding image captured at the pose is displayed as a second image 110B. The scene may correspond to a real physical environment or a virtual environment, as previously described.

[0018] like Figure 1 As shown, in most cases, the image captured at the first pose 110A is considered suboptimal because the second subject is partially obscured by the tree, and in the captured image, the faces of both subjects are not frontal. It would be desirable if the position of the camera could be automatically adjusted to obtain an image of the scene in which at least one subject is not obscured (or at least obscured by an acceptable amount) by any other object in the scene. Furthermore, it would be desirable if this image could be captured with an aesthetically pleasing composition that complies with known artistic principles (such as the rule of thirds). Reference will now be made to Figure 2 Describe a system for acquiring such images.

[0019] Figure 2 An example of a system 200 for automatically adjusting the position of a camera 201 to obtain an image of a subject in a scene is schematically shown.

[0020] System 200 includes a camera 201 operable to capture an image of a scene. Camera 201 may correspond to a real camera 201, which includes, for example, an image sensor and one or more lenses for focusing light onto the sensor. Alternatively, camera 201 may correspond to a virtual camera 201 that enables a player to capture an image of a virtual scene, for example, in a video game.

[0021] The system 200 also includes a recognition unit 202 configured to receive the image captured by the camera 201 and recognize objects of interest in the scene. The recognition unit 202 may use computer vision or machine learning techniques to perform the recognition.

[0022] The recognition unit 202 can also be configured to recognize the scene type corresponding to the captured image. In some examples, the recognition can be performed using a trained machine learning model (referred to herein as the scene type model 203). The scene type model 203 can be trained to learn high-level features representing different types of scenes. The training can be implemented using images of different scenes that have been labeled with corresponding scene identifiers (i.e., supervised learning), or using unlabeled images that are subsequently grouped into different scene types based on content similarity (i.e., using unsupervised learning). In the latter case, the developer can manually mark different groups with identifiers, indicating the scene type represented by the images in the group. In the former case, the images can be pre-labeled, for example, using metadata indicating the scene type.

[0023] High-level features may correspond to, for example, composition, color, intensity variations (or representations of these features) of an image that have been learned to correspond to a scene type. For example, in some embodiments, an image may be input to a feature representation generator such as a DenseNet, ResNet, MobileNet, etc., and a machine learning model may be trained to learn identifiers (representing scene types) corresponding to feature representations generated by the DenseNet, ResNet, MobileNet, etc.

[0024] In some examples, the scene identifier may correspond to metadata that the image used for training has been tagged with. For example, such as Instagram TM ,Flickr TM Websites such as allow users to tag images with identifiers (e.g., hashtags, geotags) indicating what scene types these images are, so these tags can be used to train scene type models so that scene type models can learn high-level image features corresponding to different labels. TM ,Twitch TM Websites such as allow users to tag videos, so video clips from these websites and their corresponding tags can be used to train the scene type model. In this way, the scene type model can learn identifiers associated with a given video clip (based on the similarity of high-level features learned for video clips of a given type).

[0025] Once high-level features of different scenes have been learned, the trained scene type model 203 can identify those scene types (if any) to which the current image most likely corresponds. Generally, it is useful to first identify the type of scene that the camera 201 is capturing, as the quality of the captured image will depend on this. For example, if the camera 201 operator intends to capture, for example, actors in a scene, the scene quality is likely to be higher if at least one of the actors' faces is visible in the scene.

[0026] It should be understood that the images used to train scene type model 203 may depend on how camera 201 is used, and the types of scenes that the operator of camera 201 may wish to be able to automatically recognize. For example, if camera 201 will be used to capture actors in a scene, scene type model 203 may be trained using movie clips associated with a movie, or the type of movie associated with the content being filmed. If camera 201 will be used to capture dynamic scenes containing a lot of action, such as sporting activities, scene type model 203 may be trained using images of such activities. Ultimately, the extent and nature to which scene type model 203 is trained may depend on the intended use of camera 201.

[0027] The system 200 also includes a posture processor 204, which is configured to obtain the posture of the object of interest in the scene relative to the camera 201. Here, the term "pose of the object" is used to describe the position and / or orientation of the object in the scene relative to the camera 201. The posture processor 204 can be configured to receive or capture data from which the posture of the object relative to the camera 201 can be determined. For example, the posture processor 204 can be configured to receive images that have been marked with posture data, which indicates the posture of the objects in these images relative to the camera that captured them. Alternatively, or in addition, the posture processor 204 can be configured to determine the posture of the object in the scene by processing the images captured by the camera. That is, the posture data can correspond to the extracted image data. Typically, the posture processor 204 is configured to obtain posture data, which indicates at least one of the direction of the object of interest relative to the camera and the distance of the object of interest relative to the camera.

[0028] In some examples, the object of interest may correspond to a person, and the 3D pose of the person may be approximated based on data received from an electronic device held or worn by the person (e.g., GPS and / or gyroscope data). For example, in a sport such as football, players may wear a sensor that tracks their position (and possibly orientation) on the field, and this data may be provided as input to a pose processor. If the pose of the camera is known (i.e., tracked), the pose of the player relative to the camera may be determined. Alternatively, known calibration techniques may be used to determine the pose of the player relative to the camera; for example, if the intrinsic characteristics of the camera are known, and the pose of the player in the 3D scene is known, the position and orientation of the camera relative to the subject may be determined.

[0029] In some examples, the pose of a subject in the scene may not be tracked using, for example, a sensor. In such examples, the pose of a subject in the scene may be determined using, for example, facial and / or skeletal tracking to detect the position and / or orientation of the subject in the scene relative to the camera. The position and / or orientation of the subject relative to the camera may also be determined using, for example, information related to known sizes and shapes of standard objects at predetermined distances and orientations from the camera. This may be applicable, for example, to situations where the camera is fixed and the subject (such as a player in a sports game) is relatively far away from the camera (e.g., video captured by a so-called "spider camera").

[0030] If there are multiple cameras, feature tracking can be used to determine the position and orientation of the camera relative to the subject. For example, if common features are detected in images captured by multiple cameras, the pose of the camera relative to the subject can be determined.

[0031] In examples where the object of interest corresponds to a virtual object in a virtual scene (eg, in a video game), gesture processor 204 may be configured to simply receive gesture data indicative of a gesture of the virtual object in the virtual scene.

[0032] It will be appreciated that any suitable method for determining the position and / or orientation of an object in a 3D scene relative to the camera 201 may be used.

[0033] Back to Figure 2 , the system 200 also includes a scene analyzer 205 that is operable to determine a scene quality associated with an image captured by the camera. The scene analyzer 205 is configured to determine the scene quality based on one or more of: pose data representing a pose of an object of interest in the scene relative to the camera (obtained as described above), an image captured by the camera at a given pose, and extracted image data (e.g., a low-pass filtered, low-resolution and / or monochrome image). These forms of scene analyzer inputs may be separate (i.e., independent) or combined, as appropriate.

[0034] The scene analyzer 205 may include a machine learning model that has been trained to map any of these inputs to a corresponding scene quality (i.e., a value or parameter) that indicates that an image of the subject captured from the current pose of the camera is likely to be considered aesthetically pleasing by a viewer. In the embodiments described herein, this model is referred to as the scene quality model 206.

[0035] The pose data and / or image data and / or extracted image data and the scene quality associated with the data may be used to train the scene quality model 206. The scene quality associated with the pose, image and / or extracted image data may be inferred from the source of the pose, image and / or extracted image data (i.e., professionally captured or popular), as described below.

[0036] In some examples, scene quality model 206 is trained to determine scene quality associated with a captured image based on at least one of:

[0037] - The subject's orientation relative to the camera (e.g., whether the subject's face is visible);

[0038] - The distance of the subject from the camera (e.g., whether the subject’s face is discernible);

[0039] - Occlusion of the subject in the captured image (e.g., what percentage of the subject is occluded, and what percentage of key areas such as the subject’s face are occluded);

[0040] - The position / placeholder of the subject within the image (aesthetic aspects such as the rule of thirds or genre-specific rules).

[0041] At least some of this information obtained from a plurality of training images for which associated scene qualities are known may be used to train the scene quality model 206. In this way, the scene quality model 206 is trained to learn the relationship between one or more of subject direction, distance, occlusion, position / occupancy, and the associated scene qualities.

[0042] It should be understood that in some cases there may be multiple subjects in the training images, and therefore the scene quality model can be trained using the direction, distance, occlusion and / or image occupancy of each of these subjects and the scene quality associated with the training images with close-ups of these subjects.

[0043] The occlusion of the subject may be determined by performing, for example, skeleton and face tracking and determining the percentage of the face or skeleton detected in the captured image that is occluded relative to a reference face or skeleton known to correspond to the subject. The face of the subject may be identified as a key area, and if the key area is occluded by more than a threshold amount, the occlusion associated with the captured image may be determined to be high. The scene quality model 206 may be trained using the occlusion associated with a plurality of training images, and the scene quality associated with those images.

[0044] It should be appreciated that in some examples, it may be desirable for the subject to not be front-facing, and thus the scene quality model may be trained using the identified scene type and the occlusions associated with images captured for that scene type. For a given image, the scene quality associated with a given occlusion may be inferred from the source or popularity associated with the image from which the occlusion has been determined. For example, occlusion information obtained from a professionally captured image may be automatically labeled as being associated with high scene quality. In this way, the scene quality model 206 may be trained to learn the relationship between the occlusions associated with a subject in an image and the scene quality associated with the image.

[0045] The position / occupancy of an object within a captured image may be determined by processing an image captured by a camera. For example, an object of interest in a captured image may be identified (e.g., by a recognition unit), and the position of the object relative to other objects / features in the captured image may be determined. In a simple example, this may involve determining whether the object of interest is approximately 1 / 3 or 2 / 3 along a horizontal axis.

[0046] It will be appreciated that the "expected" position of a subject in a scene will typically depend on the type of scene being captured, and so the scene quality model may be trained using the scene type (e.g., as an identifier) ​​and the position of the subject in images of that scene type. In some examples, the recognition unit may be configured to identify the relative position / occupancy of a subject in an image, and this information may be provided as input to the scene quality model, along with the scene quality associated with the image from which the relative position / occupancy of the subject in the image was obtained. Again, if the position / occupancy of the subject is obtained from professionally captured and / or popular content, the scene quality associated with the position / occupancy of the subject in the image may be identified as high. In this way, the scene quality model 206 may be trained to learn the relationship between the position / occupancy of a subject in images of a given scene type and the associated scene quality.

[0047] In some examples, the scene quality model can be trained using pose data captured during the recording of a live event. For example, during the broadcast of an event such as a sports game (e.g., football, basketball, rugby, ice hockey, etc.), the positions of the players on the field can be tracked, and the pose of the camera can also be tracked (or obtained based on, for example, known field geometry and known player positions). This pose data can be input to the scene quality model, and the pose data can be marked as corresponding to high scene quality because it corresponds to content captured in a professional manner. In this way, the scene quality model is able to learn camera-relative player poses that correspond to high scene quality.

[0048] In some examples, pose data obtained from video game clips can be used to train a scene quality model. In such an example, the pose of a virtual camera relative to a virtual object (e.g., a player's avatar) can be accurately known, and the pose data can be used to train a scene quality model. The scene quality associated with the pose data can be identified as high if, for example, it corresponds to a video clip captured by a popular content creator or has a high popularity associated with it (e.g., a large number of online "views" or "likes"). By training the scene quality model 206 with this data, the scene quality model 206 can learn which poses of the virtual camera relative to the virtual object may result in the capture of an image with high scene quality. Similarly, it may be desirable to train the scene quality model using scene types associated with the virtual camera pose data to ensure that the relationship between the virtual camera pose (relative to the subject) and the scene quality is learned on a per-scene type (i.e., per-game type) basis.

[0049] In some examples, the scene quality model 206 can be further trained using training images of different objects of interest in different scenes. That is, images can be provided as input to the scene quality model as opposed to pose, occlusion, and composition information extracted from these images. In this way, the scene quality model can be trained to implicitly learn image features (e.g., lighting, occlusion, composition, etc.) that correspond to images of high scene quality. As described above, high scene quality can be inferred from the source of the training images, for example, if they correspond to professionally captured content or content that is known to be popular. For example, from Instagram TM , Flickr TM 、500px TM ,,Youtube TM Images acquired by the method may be known to have high scene quality if they have a large number of "likes" or "views" associated with them. Training images may be labeled as having high scene quality based on at least one of their source, associated popularity (e.g., likes or views exceeding a threshold), and creator.

[0050] It should be understood that there may be variability in the quality of the training images used to train the scene quality model 206. For example, the quality of a broadcast sports clip may be higher than that of a YouTube video clip. TM To compensate for this, the training images can be converted to a standard format using, for example, an external system such as CycleGAN.

[0051] By training the scene quality model using this data, the scene quality model can learn the high-level image features (color, composition, lighting, etc.) for a given scene type that are common to high scene quality images. Once trained, the scene quality model can determine whether the currently captured image of a given scene type has high scene quality.

[0052] The system 200 also includes a controller 207 configured to receive input from the scene analyzer 205 and, in response thereto, adjust the pose of the camera 201. The input provides an indication of whether a scene quality associated with a current image is less than a threshold.

[0053] In embodiments where the camera 201 is a real camera 201, the controller 207 may be configured to generate a signal received by a motion device 209 (such as a motor of a device into which the camera 201 is incorporated or connected). The motion device 209 may be operable to control at least one of a translation and a rotation of the camera 201. In some examples, the camera 201 may be incorporated into a device including wheels and / or propellers (e.g., a drone), and the controller 207 may be configured to control the power applied to the wheels and / or propellers (and any steering devices associated therewith). In embodiments where the camera 201 is a virtual camera 201, the controller 207 may simply determine a new pose in the virtual scene to which the virtual camera 201 is to move.

[0054] Optionally, if the camera is a handheld camera, the controller can adjust the camera's posture by controlling a guidance interface on the camera, rather than by controlling a motor, such as a graphic of an arrow pointing in the desired direction of movement on the camera's rear screen (and / or in the viewfinder), optionally indicating the desired distance in the direction shown, such as a number or line that decreases as the user moves the camera to the desired position.

[0055] In some examples, controller 207 includes a machine learning model (referred to herein as pose model 208) that is trained to determine poses of camera 201 that are more likely to result in capturing an image of an object of interest with higher scene quality. Pose model 208 may include an agent trained using deep reinforcement learning. For example, the agent may be trained by moving around a scene randomly or predictively to learn poses that result in the greatest overall scene quality for objects in the scene. That is, the reward function may correspond to an average scene quality that varies over time. As previously described, scene quality model 206 may be used to determine the scene quality of a captured image.

[0056] In a preferred example, the agent is trained by moving around a virtual scene including virtual objects (this is preferred because movement of the camera around the subject will be easier to achieve). For example, the virtual scene can be provided by a game engine, and the virtual scene can correspond to a real scene where a real camera is to be deployed. For each pose of the virtual camera relative to the subject in the virtual scene, the scene quality associated with the pose can be determined using the trained scene quality model 206, and the agent can be trained to learn new poses that may result in capturing images with higher associated scene quality. In some examples, the virtual scene can include multiple virtual subjects, and the scene quality model can be configured to determine the scene quality based on the poses of these subjects relative to the camera.

[0057] In one example, the inputs to the pose model 208 may include the position and / or orientation of the virtual camera, the position and / or orientation of the virtual subject, and the output of the scene quality model 206. The pose model may be trained using these inputs to learn (via deep reinforcement learning) a pose of the virtual camera relative to the virtual subject that maximizes average scene quality.

[0058] The agent can be trained by moving around a number of different scenes and capturing images of different objects of interest in those scenes. Once the agent has been trained for a sufficient number of different scene types and different subjects, the gesture model 208 can be used to adjust the position of the camera 201. The gesture model can also be trained to learn one or more camera parameters that result in capturing images of higher scene quality, as described later.

[0059] In some examples, the objects of interest (i.e., subjects) may include human characters. In such examples, the scene quality model 206 may be trained using pose and / or image data associated with the human characters. In these examples, the pose model may be trained to adjust the pose of the camera to ensure that less of the human subject's face is occluded (e.g., facing more forward).

[0060] It should be understood that for human subjects, the recognition unit 202 can be configured to detect human subjects in the scene (using, for example, image segmentation, facial recognition, etc.) and provide them to the scene quality model 206 as input so that the scene analyzer 205 can use an appropriately trained model (i.e., a model trained with images of human subjects in the corresponding scene) to determine the scene quality of the captured image.

[0061] Typically, for a scene containing multiple characters, there will be one or more primary characters and one or more secondary characters. If the camera 201 operator is capturing characters in the scene, the primary actor may correspond to the character that is speaking or performing a predetermined action. Typically, it is desirable for the camera 201 to focus on the primary actor, so the captured image of the scene should include more content of the primary actor than the secondary actor. In addition, it is typically desirable for the primary actor's face to not be obscured.

[0062] Thus, in some examples, scene quality model 206 may be trained using poses and / or image data associated with multiple characters, where one or more characters in an image are identified as a primary character. Image data and / or pose data corresponding to a primary character that is visible (and / or occluded by less than a threshold amount) in a captured image may be labeled as having high scene quality, such that scene quality model 206 is trained to learn that images that are close-up of the primary character are of higher quality than images in which the primary character is not visible. Thus, controller 207 may be trained to determine poses for camera 201 that are more likely to result in an image of a scene in which the primary character's face is not occluded, or at least appears in a manner appropriate for the type of scene that the operator of camera 201 intends to capture.

[0063] In some examples, a user may provide input indicating which characters are primary characters, and the scene quality model 206 (after sufficient training) may determine the scene quality of the scene based on this identification. This identification may be performed using, for example, a touch screen displaying a video captured by the camera 201, and the user may select areas of the touch screen corresponding to primary characters in the scene. More generally, the identification unit 202 may be operable to receive input indicating one or more primary characters in the scene.

[0064] In some examples, detection of a primary character in a scene may be performed automatically. For example, the system 200 may include one or more microphones for detecting a character's voice, and based on the detection of the character's voice, the character may be identified as the primary character in the scene. Alternatively, or in addition, the times at which different characters in the scene are set to speak may be known in advance based on, for example, a script, and this information may be provided to the recognition unit 202 to identify the different characters on which the camera's focus should be when capturing the scene. Typically, the system may include an audio unit (not shown) that is operable to receive voice data indicating that at least one character is or is about to begin speaking, and this information may be provided to the recognition unit 202, which then determines which character corresponds to the primary character.

[0065] In some scenes, there may be multiple main characters or at least two characters that are intended to be captured with camera 201. In such examples, at least two characters may be identified for inclusion in a captured image (by any of the above-described means), and scene quality model 206 may be configured to determine a scene quality associated with the captured image based on the identification of multiple characters in the scene. Again, scene quality model 206 may be trained using pose data and / or image data associated with multiple main characters, and scene quality may be higher for images in which faces of the main characters are visible. Thus, controller 207 may be configured to determine a pose of camera 201 that results in minimal occlusion of faces of the multiple main characters.

[0066] In some examples, the camera 201 operator may wish to include other objects of interest (i.e., other than people) in the captured images. In such examples, the recognition unit 202 may be configured to receive input indicating one or more objects of interest to include in the captured images. The input may be provided to the scene quality model 206, which then determines the scene quality of the captured images based on the recognized objects, the poses of these objects relative to the camera 201, and / or the captured images of the scene.

[0067] Objects may be identified for inclusion in a captured image by user input. For example, a video image captured by camera 201 may be displayed on a screen, and a user may select an area of ​​the screen corresponding to an object of interest that they intend to include in the captured image. Where machine learning or computer vision techniques may be used to identify different objects in a scene, identification of objects may be automated to some extent, with an operator providing manual input as to which identified objects have a higher priority for inclusion in the captured image.

[0068] In some examples, an operator may wish to maximize the number or amount of particular objects of interest in a captured image. For example, in horror movies, characters are typically shown surrounded by trees to evoke fear. Thus, the camera 201 operator may provide input indicating that the number of trees in a captured image is to be maximized. For example, this may involve selecting, for example, "trees" as an object of interest in a displayed image, and providing further input to maximize the number of trees in the captured image. These inputs may be provided to the scene quality model 206, which then determines the scene quality of the captured image based on the number of objects of interest identified in the captured image, any identified characters, the captured image, and / or the pose of any identified characters relative to the camera 201. It should be understood that the scene quality will generally be higher for images where identified characters are visible, so this ensures that the camera 201 is not moved as far as possible when trying to maximize the number of, for example, trees in the capture.

[0069] For some content, it may be necessary to adjust the position of the camera 201 based on the position of the audio source in the scene (not necessarily associated with a character). Therefore, in some examples, the system 200 may include two or more microphones for detecting the audio source in the scene. The recognition unit 202 may be configured to receive an audio signal detected by the microphone, and in response thereto, identify a known audio source corresponding to the detected audio signal. This can be achieved using, for example, machine learning, or by simply comparing the spectral characteristics of the audio signal with the spectral characteristics of multiple known audio sources. The gesture processor 204 may be configured to determine the 3D gesture of the audio source relative to the microphone based on the audio signal detected by the two or more microphones. In a preferred example, two or more microphones are associated with a device including the camera 201, so that the gesture of the audio source relative to the camera 201 can be inferred from the audio signal detected by the microphone.

[0070] The scene quality model 206 may be configured to receive an indication of an identified audio source (i.e., type) and a 3D pose of the audio source relative to the camera, and in response thereto, determine a scene quality associated with the captured image. This may be a subject in addition to any other object of interest, such as a human character, identified as an object of interest by the recognition unit 202. It will be appreciated that in such an example, the scene quality model 206 will be further trained using data indicating different types of audio sources in the scene, and pose data indicating at least one of the position and orientation of the audio source relative to the camera 201 used to capture the subject in the scene.

[0071] As such, scene quality model 206 may further determine scene quality associated with a captured image based on pose data and / or image data associated with an image subject, and a pose of an audio source relative to camera 201 .

[0072] In one example, scene quality model 206 may be trained using video frames in which a character is running away from an explosion, and scene quality model 206 may learn the pose of camera 201 that should be adopted when capturing such a scene (e.g., based on the pose of the actor relative to the camera, the captured image, and the location of the explosion).

[0073] In some examples, the scene may be a virtual scene, and therefore the type of audio source and its relative position in the scene may be accurately known. In addition, the time at which different audio sources generate audio may also be known. Therefore, the type of audio, relative position, and optionally time information may be input into the scene quality model 206. The scene quality model 206 may then determine the scene quality associated with the current captured image of the virtual scene based on, for example, the captured image, the position of the player avatar, and any audio source in the scene. Again, in such an example, the scene quality model 206 may be trained using training images (e.g., video frames), and / or, for example, the 3D pose of the player avatar, and the relative position of any audio source in the scene. It should be understood that in some video games, there may be multiple audio sources, and therefore the recognition unit 202 may be operable to detect the audio source that contributes most to the player's experience (e.g., the loudest and or closest audio source).

[0074] In some examples, the scene quality model 206 can be further trained using intrinsic camera data, which indicates one or more camera parameters used to capture images of the subject. The intrinsic data can include, for example, the focal length, aperture, shutter speed, any special modes (e.g., HDR) of the camera used to capture the training images. For example, if the scene quality model is trained using images of real scenes captured by a real camera, the captured images can be used as metadata to encode the intrinsic camera data. In this way, the scene quality model 206 can be trained to determine the relationship between the relative pose of the subject with respect to the camera (optionally, the captured image data), one or more camera parameters, and the corresponding scene quality.

[0075] In these examples (where the intrinsic parameters are taken into account), the controller 207 may be further configured to adjust one or more intrinsic parameters of the camera 201 based on the determination that the scene quality of the current image is less than a threshold. For example, the controller 207 may be configured to adjust the focal length (i.e., zoom), aperture, shutter speed, etc. of the camera 201 based on the determined scene quality. Adjustments to these parameters may be determined based on the determined scene quality output by the scene quality model 206.

[0076] As previously described, the controller 207 may include a pose model 208 trained to determine adjustments to the pose of the camera 201. In examples where one or more camera parameters are also adjusted, the pose model may be further trained to determine the one or more camera parameters to maximize scene quality. This may entail, for example, using deep reinforcement learning to train an agent to capture multiple images of a subject in a virtual scene, and learn the combination of camera parameters that results in the captured images having the highest scene quality (for a given position). The trained scene quality model 206 may be used to determine the scene quality of the images captured during this training (i.e., by determining how close these images are to images of corresponding subjects in corresponding scenes that are known to have high scene quality).

[0077] In additional or alternative examples, the controller may be configured to adjust, for example, the depth of field based on the pose that the camera has been moved to. For example, if the camera has been moved to a position less than a threshold distance from the subject, the controller may determine that the subject of the image should be in focus with a blurred background. Conversely, if it is determined that the distance between the subject in the image and the adjusted pose of the camera is greater than the threshold distance, the controller may be configured to control, for example, the aperture of the camera to ensure that all or a large portion of the scene is in focus.

[0078] Although the above examples have been described as separate units within the system, it should be understood that one or more of them may be included in the same device and the device itself constitutes a camera device. The camera device may correspond to a physical device with a controllable position and orientation. For example, the camera device may correspond to a camera device with powered wheels or propellers (e.g., a drone). In examples where all components are incorporated into the same physical device, various machine learning models can be trained using, for example, one or more servers (e.g., the cloud), and the trained machine learning models can be exported to relevant modules of the physical camera device.

[0079] Thus, in some embodiments, a system is provided in which a real camera device comprises: a sensor operable to capture an image of a scene; and one or more selected from a list comprising: an identification unit (e.g., operable to identify an object of interest in the captured image); a posture processor (e.g., operable to obtain a posture of an object of interest in the scene relative to the camera); a scene analyzer (e.g., configured to detect scene quality associated with an image captured by the camera, the scene analyzer comprising a machine learning model trained to determine scene quality associated with the captured image); and a controller (e.g., operable to control a posture of the camera device based on a determination that the scene quality of the current image is less than a threshold).

[0080] It should be understood that the machine learning models described herein may include, for example, at least one of a trained neural network, such as a trained convolutional or recurrent neural network (CNN, RNN, respectively), a multi-layer perceptron (MLP), or a restricted Boltzmann machine. Ultimately, any suitable machine learning system may be used.

[0081] It should also be understood that in some examples, the camera need not capture images to determine scene quality. For example, in some examples, scene quality may be determined based solely on the pose of the subject relative to the camera without actually capturing any images (e.g., if the position of the subject and camera is being tracked during a broadcast event).

[0082] It should also be understood that the methods described herein may be performed on conventional hardware appropriately adapted by software instructions or by inclusion or replacement of dedicated hardware (as applicable). Thus, the required adaptation of existing portions of conventional equivalent devices may be implemented in the form of a computer program product comprising processor-implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, PROM, RAM, flash memory, or any combination of these or other storage media, or implemented in hardware as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array) or other configurable circuit suitable for adapting conventional equivalent devices. Additionally, such a computer program may be sent via a data signal on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

Claims

1. A system for capturing an image, comprising: a camera operable to capture an image of a scene; a recognition unit configured to recognize an object of interest in an image of the scene; a pose processor configured to obtain a pose of the object of interest in the scene relative to the camera; a scene analyzer operable to determine, based on at least one of the obtained poses of the object of interest and the images captured by the camera, a scene quality associated with the images captured by the camera at the respective poses; wherein the scene analyzer comprises a first machine learning model, the first machine learning model being trained to determine the scene quality associated with images captured by the camera at various poses, and identifying poses of high scene quality by analyzing professionally captured images or images with high popularity on a website; and a controller configured to cause a pose of the camera to be adjusted based on a determination that the scene quality of an image captured at a current pose is less than a threshold; Wherein, the recognition unit is configured to recognize the scene type corresponding to the image captured by the camera; wherein the scene analyzer is further configured to determine scene quality associated with images captured by the camera at various postures based on the identified scene type; and In which, the recognition unit includes a third machine learning model, which is trained to identify the scene type corresponding to the image captured by the camera, and the third machine learning model is trained using images of different scene types and corresponding scene identifiers.

2. The system according to claim 1, wherein: The first machine learning model is trained using pose data indicating a pose of the object of interest in the scene relative to the camera that captured the image, and / or training images of the object of interest, wherein the pose data and / or training images are labeled with respective scene qualities.

3. The system according to claim 2, wherein: The gesture data and / or training images are marked as having high scene quality based on at least one of a source of the gesture data and / or training images and user feedback associated with the gesture data and / or training images.

4. The system according to claim 2 or 3, wherein: The controller is configured to adjust one or more intrinsic parameters of the camera based on a determination that the scene quality of the image captured at the current pose is less than a threshold; and Wherein, the first machine learning model is further trained using intrinsic data indicating one or more intrinsic parameters of the camera for which the posture data and / or training images were obtained.

5. The system according to claim 1, wherein: The controller includes a second machine learning model that is trained to determine a pose of the camera that may result in capturing an image of the object of interest with higher scene quality.

6. The system according to claim 5, wherein: The second machine learning model comprises an agent trained using deep reinforcement learning; and Therein, the agent is trained to learn poses that maximize the scene quality of images captured by the camera, the agent being trained by moving around a plurality of different virtual scenes and capturing virtual images of objects of interest in those scenes.

7. The system according to claim 6, wherein: The second machine learning model is configured to determine a scene quality associated with the virtual image by inputting the virtual image into the trained first machine learning model.

8. The system according to claim 1, wherein: The object of interest includes a character in a scene, and wherein the first machine learning model is trained using pose data and / or training images of the character in the scene.

9. The system according to claim 8, wherein: The object of interest includes multiple roles, and the system includes: an input unit operable to receive input from a user, the input indicating one or more characters to be included in the captured image; and Therein, the scene analyzer is configured to determine a scene quality based on a 3D pose of the character to be included in the captured image and / or the image captured by the camera.

10. The system according to claim 9, comprising: an audio unit operable to receive voice data indicating that at least one of the characters is speaking or is about to speak; wherein the recognition unit is configured to recognize at least one of the characters as a main character based on the voice data; and The scene analyzer is configured to detect the scene quality based on a posture of the at least one main character in the scene relative to the camera and / or the image captured by the camera.

11. The system according to claim 1, wherein: The recognition unit is configured to recognize a source of audio in the scene, and the gesture processor is configured to determine a gesture of the source of the audio relative to the camera; and The scene analyzer is further configured to determine a scene quality associated with the captured image based on a detected pose of a source of the audio relative to the camera.

12. The system of claim 1, wherein: The camera is a real camera, and the system comprises a motion device for controlling the position and / or orientation of the camera; and Wherein, the motion device comprises at least one of a wheel and a propeller, and the motion device is arranged to receive input from the controller.

13. The system of claim 1, wherein: The camera comprises: a sensor operable to capture an image of a scene; and One or more selected from a list consisting of: i. the identification unit, ii. the gesture processor, iii. the scene analyzer, and iv. said controller.

Images