Techniques and Apparatus for Implementing a Camera Manager System Capable of Generating Frame Suggestions from a Frameset - Patent application

Abstract

Techniques and apparatus are described for implementing a camera manager system capable of generating frame suggestions from a frame set. The camera manager system (120) utilizes at least one of a facial diversity scorer (406) and an aesthetic diversity scorer (408) in conjunction with a temporal diversity scorer (404) to select and suggest diverse frames from the frame set (304). In this manner, the camera manager system (120) enables a computing device (102) to provide a user (10) of the computing device with a better selection of suggested frames. The better selection of suggested frames enables the computing device (102) to improve the quality of the user experience when using the computing device and / or the computing device's camera application. The better selection of suggested frames further reduces wasted resources (e.g., similar image storage, processor usage for processing the capture of additional frames, battery usage associated with capturing additional frames, etc.).

Description

[Background technology]

[0001] background Computing devices (e.g., smartphones) that include image capture applications often include an element that captures and suggests one or more frames captured by the device after a user presses a physical button or a shutter button on the device's graphical user interface (GUI). Because current frame suggestion techniques select frames based on frame quality metrics, the computing device often suggests a large number of visually similar frames to the user. The presentation of such visually similar frames can be of limited use to the user and can degrade the user experience associated with the use of frame suggestion techniques. Summary of the Invention [Problem to be solved by the invention]

[0002] overview This specification describes techniques and devices for implementing a camera manager system capable of generating frame suggestions from a set of frames (e.g., images, photos, photographs, videos). In one aspect, the camera manager system utilizes at least one of a facial diversity scorer and an aesthetic diversity scorer, along with a temporal diversity scorer, to select and suggest diverse frames from the set of frames. By doing so, the camera manager system saves power, improves accuracy, and / or reduces latency compared to many common techniques and devices for frame suggestions. The camera manager system further provides a better user experience. [Means for solving the problem]

[0003] The method described herein comprises receiving an image data stream defining a first frame and a set of frames not including the first frame, and then performing a frame score generation process to calculate a frame diversity score. The frame score generation process includes calculating a temporal diversity score for the frames of the frame set relative to the first frame, calculating a facial diversity score for the frames of the frame set relative to the first frame, and calculating an aesthetic diversity score for the frames of the frame set relative to the first frame. The frame diversity score for the frames of the frame set relative to the first frame is calculated based on the facial diversity score, the aesthetic diversity score, and the temporal diversity score. The frame score generation process further includes using the frame diversity score to determine whether to include the first frame as part of an image object representing a proposed frame of the image data stream. Such a method may result in improved power savings, improved accuracy, and / or reduced latency compared to many common techniques and devices for frame proposal. The camera manager system further provides a better user experience.

[0004] This specification also describes computer readable media having instructions for carrying out the methods summarized above and other methods described herein, as well as apparatus and means for carrying out these methods.

[0005] This Summary is provided to introduce a simplified concept of techniques and apparatus implementing a camera manager system capable of generating frame suggestions from a frameset, which are further described below in the detailed description and drawings. This Summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to define the scope of the claimed subject matter.

[0006] One or more aspects of techniques and apparatus for implementing a camera manager system capable of generating frame suggestions from a frame set are described in detail herein with reference to the following drawings, in which like numbers are used throughout to reference like features and components: [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram illustrating an example environment in which techniques for implementing a camera manager system capable of generating frame suggestions from a frame set can be practiced. [Diagram 2] FIG. 1 is a schematic diagram illustrating an example implementation of a computing device including a camera manager system capable of generating frame suggestions from a set of frames. [Diagram 3] FIG. 2 is a block diagram illustrating how components of a camera manager system may be integrated with or within a camera application of a computing device according to an example implementation. [Figure 4] FIG. 2 is a block diagram illustrating components of a feature scoring module of the camera manager system according to an example implementation. [Diagram 5] FIG. 1 illustrates an exemplary method for implementing a camera manager system capable of generating frame suggestions from a set of frames. [Figure 6] FIG. 1 illustrates another exemplary method for enabling a camera manager system capable of generating frame suggestions from a set of frames. [Figure 7] A diagram showing various components of an exemplary computing device that can be implemented as any type of client, server, and / or electronic device as described with reference to Figures 1-6 for implementing a camera manager system capable of generating frame suggestions from a frame set, or in which techniques enabling such a camera manager system may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Detailed Description Summary This specification describes aspects of techniques and apparatuses for implementing a camera manager system capable of generating frame suggestions from a set of frames (e.g., images, photos, photographs, videos). The camera manager system may utilize at least one of a facial diversity scorer and an aesthetic diversity scorer along with a temporal diversity scorer to select and suggest diverse frames from the set of frames. In this manner, the camera manager system enables a computing device to provide a better selection of suggested frames to a user of the computing device. The better selection of suggested frames further reduces wasted resources (e.g., similar image storage, processor usage to process the capture of additional frames, battery usage associated with the capture of additional frames, etc.). With the better selection of suggested frames, the camera manager system can improve the quality of the user experience when using the computing device and / or the camera application of the computing device.

[0009] In an exemplary use, assume that a user uses a smartphone camera application to take multiple photographs (frames) of a scene (e.g., a group of the user's friends posing in front of an architectural piece). The user may activate the camera application to capture a frame by pressing a shutter button (e.g., a physical button, a user interface button). Upon reviewing the captured frame, the user realizes that the eyes of one of the user's friends were closed when the frame was captured and that the frame is not satisfactory to the user and / or the frame needs to be retaken.

[0010] In addition to capturing a frame associated with the moment the user presses the shutter button, the camera application may also capture a number of additional frames before and / or after the shutter button is pressed. The smartphone may then present a diverse selection of frames, i.e., the captured frame and the additional frames, to the user as the user reviews the captured frames. By presenting the user with a diverse selection of frames taken before and after the shutter button was pressed, the likelihood of capturing an acceptable image is increased. In presenting the user with a temporally diverse selection of frames, the user may not be able to determine whether a given frame is better or worse than other frames, including the frame captured when the shutter button was pressed. This may result in wasted smartphone resources, such as, for example, storage of too many similar images, excessive processor usage to process the capture of the additional frames, and excessive battery usage associated with the capture of the additional frames. The result may be a less than optimal user experience. Such wasted resources may be problematic in computing devices such as smartphones, where data storage and battery size may be limited by the size of the smartphone.

[0011] In contrast, consider the disclosed techniques and apparatus that implement a camera manager system capable of generating frame suggestions from a set of frames. In an aspect, the camera manager system utilizes one or more diversity scorers (e.g., a facial diversity scorer, an aesthetic diversity scorer, a temporal diversity scorer) in the frame suggestion process. By utilizing the diversity scorers, the camera manager system determines a more diverse selection of frames to present to the user such that the proposed frames are visually distinct, thereby reducing wasted resources (e.g., storage of similar images, processor usage to process the capture of additional frames, battery usage associated with the capture of additional frames, etc.) and improving the quality of the user experience. Providing a more diverse selection of frames may enable a user to analyze the frame selection more quickly and / or efficiently, thereby aiding the user in, for example, image analysis or image classification tasks. The user need only use the computing device for a shorter period of time, which may reduce battery and processor usage of the computing device. The user may avoid being forced to capture additional frames, again reducing battery and processor usage of the computing device.

[0012] This is just one example of how the described techniques and apparatus implementing a camera manager system capable of generating frame suggestions from a set of frames can be used to determine a more diverse selection of frames to be presented to a user. Other examples and implementations are described throughout this specification. This specification next describes an example operating environment, followed by example apparatus, methods, and systems.

[0013] Operating environment FIG. 1 illustrates an example environment 100 in which techniques implementing a camera manager system capable of generating frame suggestions from a frame set may be utilized. The example environment 100 includes an example implementation of a computing device 102 capable of executing techniques implementing a camera manager system capable of generating frame suggestions from a frame set. In FIG. 1, a user 10 is shown holding and operating the computing device 102 to capture an image, for example, using a camera application. Both a back view 102-1 and a front view 102-2 of the computing device 102 are illustrated. Although the computing device 102 in FIG. 1 is illustrated as a smartphone, in other aspects the computing device may be another type of computing device (e.g., a tablet, a laptop, a camera, a desktop computer, a computing watch, a gaming system, a computing eyeglasses, a home automation and control system, a smart appliance, an automobile, a television, an entertainment system, an audio system, a drone, a trackpad, a drawing pad, a netbook, an e-reader, a home security system, etc.). It should be noted that in aspects, the computing device 102 may be wearable, non-wearable but mobile, or relatively fixed (e.g., desktop and appliance). Figure 2 further illustrates the computing device 102 of Figure 1.

[0014] The computing device 102 comprises or is associated with a camera system 104 including at least one image capture device 106 (e.g., camera), at least one display 108 (e.g., display screen, display device), one or more computer processors 110 (processor(s) 110), and a computer-readable medium 112 (CRM 112). The computing device 102 may communicate with the image capture device 106 for capturing images and / or videos. As shown in FIG. 1, the computing device 102 may include at least one built-in or internal image capture device 106 (e.g., camera, charge-coupled device (CCD)). In another exemplary implementation (not shown), the image capture device 106 may be external to the computing device 102 and may be in communication with the computing device, for example, via a direct connection or wireless coupling.

[0015] The display 108 may include any suitable display device (e.g., a touch screen, a liquid crystal display (LCD), a thin film transistor (TFT) LCD, an in-plane switching (IPS) LCD, a capacitive touch screen display, an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode (AMOLED) display, a super AMOLED display). The display 108 may be combined with a touch-sensitive or presence-aware input device to form a touch-sensitive or presence-aware display for receiving user input from a stylus, finger, or other means of gesture input. The display 108 may display graphical images and / or instructions provided by the computing device 102 to assist a user in interacting with the computing device 102. The display 108 may be separate from the camera system 104 (as shown in FIGS. 1 and 2) or may be part of the camera system 104 (not shown as such).

[0016] The display 108 presents a GUI of an application 114 (e.g., a camera application 114). The GUI of the application 114 may include one or more input controls (e.g., a GUI shutter button 116) for providing input to the computing device 102, e.g., for triggering the capture of an image. Thus, the application 114 may receive user input, e.g., actuation of the shutter button 116, via the presence-aware display 108. The computing device 102 may also include input / output (I / O) devices 122, e.g., one or more physical buttons 118 (shown in FIG. 1 ). The I / O devices 122 are for providing input to the computing device 102 (e.g., for triggering the capture of an image). The application 114 may be embodied in one or more of software, applets, firmware, peripherals, hardware, or other entities configured to operate the image capture device 106. In one example, the application is a camera application 114 installed on the computing device 102. In another example, the application is part of an operating system (OS) that enables camera functionality in the application.

[0017] The CRM 112 may include any suitable memory or storage device, such as random-access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), non-volatile RAM (NVRAM), read-only memory (ROM), or flash memory. The CRM 112 may include a memory system. The CRM 112 includes device data. The device data may be user data, multimedia data, a ring buffer, a candidate buffer, a feature store, application(s) 114, a camera manager system (camera manager 120), a feature extraction module, a feature scoring module, a frame selection module, a machine learning model (e.g., a score model), and / or an operating system (not shown) of the computing device 102, which are implemented as computer-readable instructions on the CRM 112 executable by the processor(s) 110 to provide some or all of the functionality described herein. For example, the processor(s) 110 can be used to implement the disclosed techniques and apparatus to implement a camera manager system 120 (camera manager 120) capable of executing instructions on the CRM 112 to generate frame suggestions from a frame set.

[0018] The device data may include executable instructions for a camera manager 120 executable by the processor(s) 110. The camera manager 120 represents functionality that causes the computing device 102 to perform operations described herein to generate frame suggestions from a set of frames captured by the camera system 104. The operations may include receiving input from a user providing input, for example, by pressing a physical button 118 or by pressing a shutter button 116 on a GUI of the application 114. The device data may further include executable instructions for one or more modules (e.g., a feature extraction module, a frame selection module, a result generator module) executable by the processor(s) 110 to implement the camera manager system.

[0019] Various implementations of the disclosed systems and apparatus implementing a camera manager system capable of generating frame suggestions from a frame set may include a system-on-chip (SoC), one or more integrated circuits (ICs), a processor with embedded processor instructions or a processor configured to access processor instructions stored in memory, hardware with embedded firmware, a printed circuit board with various hardware components, or any combination thereof.

[0020] These and other capabilities and configurations, and the manner in which the entities of Figures 1 and 2 act and interact, are described in more detail below. These entities may be further divided, combined, etc. The environment 100 of Figure 1, the computing device 102 of Figures 1 and 2, and the detailed illustrations of Figures 2-4 illustrate some of the many possible environments and systems in which the described techniques may be employed. Figures 5 and 6 illustrate some of the many possible ways in which techniques implementing a camera manager system capable of generating frame suggestions from a frame set may be implemented. Figure 7 illustrates aspects of techniques and systems implementing a camera manager system capable of generating frame suggestions from a frame set in the context of the computing device 102 of Figures 1 and 2, although, as noted above, the applicability of the features and advantages of the described techniques and apparatus are not necessarily so limited, and other implementations, including other types of electronic devices, may be within the scope of the teachings of this disclosure.

[0021] In an aspect, image data may be collected by sampling frames from an available camera stream of image data to define a frame set. For example, the camera application 114 of the computing device 102 may present a live preview based on the image data stream from the image capture device 106. A plurality of frames defining a frame set may be sampled from a corresponding live stream of image data. Utilizing the disclosed techniques for generating frame suggestions from a frame set, a subset of a diverse selection of frames may be saved for later presentation. The camera manager system (camera manager 120) may initiate the capture of a frame stream by the image capture device 106 without prompting from a human user. Thus, a frame stream may be acquired even if a live preview is not presented to the user 10 by the camera application 114.

[0022] The camera manager 120 may be launched in response to the camera system 104 and / or the camera application 114 being launched or active on the computing device 102. Also, the camera manager 120 may be launched in response to an image stream becoming available. For example, the camera manager 120 may be launched when the live preview of the camera application 114 is activated. Furthermore, the camera manager 120 may be launched in response to a user interaction. For example, the camera manager 120 may be launched by a user input on a shutter button (e.g., the shutter button 116 of the camera application 114, the physical shutter button 118) and image data that defines a new frame (e.g., a shutter frame) that has been collected. Also, in response to a second user input, the camera manager 120 may be stopped. Thus, a summary of the photos may be generated that includes the content that was missed between the manual capture of the two photos. For example, the camera manager 120 may be launched or stopped by a dedicated button or a user interface (UI) widget.

[0023] system 3 is a block diagram illustrating an architecture 300 utilized in a technique for generating frame suggestions from a frame set, according to an example implementation. For example, it illustrates how components of a camera manager 120 may be integrated with or within a camera application. When a user (e.g., user 10) presses a shutter button (e.g., shutter button 116, shutter button 118) on a computing device (e.g., computing device 102), an image stream (e.g., camera stream 302) is generated.

[0024] The image frames 304 (frames 304) may be sampled from an available image stream, for example, a camera stream 302 generated by the camera system 104 of the computing device 102 in a camera preview mode or a capture mode. Examples of the camera stream 302 include an HD stream (1024×768) and a RAW stream (4032×3024). In the camera preview mode, an application (e.g., camera application 114) may provide a live preview to a user (e.g., user 10) on a display (e.g., display 108) based on the camera stream 302. The frames 304 may include a selected frame 304b and a first frame 304a. In an aspect, the first frame 304a is the frame 304 with the most recent timestamp.

[0025] The architecture 300 includes at least one machine learning model trained to receive one or more types of input data (e.g., one or more features associated with a case or example) and, in response, provide one or more types of output data (e.g., one or more predictions). For example, one or more score models 305 may subscribe to a camera stream 302 and receive as input an image frame 304 from the camera stream 302. The score model 305 may then output a score for the frame. In an embodiment, the score model 305 is a face quality score model used to calculate and output a face quality score representative of the face features of the frame. The face quality score may be calculated as a weighted linear combination of one or more face attributes (e.g., eyes open, mouth open, staring forward, smiling, amused, pleased, elated, surprised). In an embodiment, the score model 305 is an aesthetic value score model used to calculate and output an aesthetic value score representative of the scene-related features (e.g., non-facial features) of the frame. The scene-related features may include broad spatial information related to aesthetics (e.g., object layout, blur, camera focus). The camera manager system 120 may use the score output by the score model 305 for the frame.

[0026] A machine learning model may be or may include one or more artificial neural networks (also referred to simply as neural networks). A neural network may be organized into one or more layers, e.g., an input layer, an output layer, and one or more hidden layers located between the input layer and the output layer. The one or more neural networks may be used to provide embeddings based on input data. For example, an embedding may be a representation of knowledge abstracted from the input data into one or more learned dimensions. In some implementations, embeddings may be extracted from the output of the network, while in other cases, embeddings may be extracted from any hidden node or hidden layer of the network (e.g., a bottleneck layer of the network, a layer near to but not the final layer of the network). A bottleneck layer is utilized to create a constriction in the network that contains fewer nodes compared to previous layers of the model and reduces the dimensionality of the embeddings.

[0027] The camera manager system 120 may extract embeddings (results) from the bottleneck layer of the score model 305. Such embeddings may include one or more of facial expression embeddings (e.g., facial expressions in a frame), face location embeddings (e.g., face location in a frame), face count embeddings (e.g., number of faces in a frame), or aesthetic embeddings (e.g., object layout embeddings). The extracted embeddings may capture at least one of global spatial information (e.g., layout) or fine-grained detailed differences (e.g., facial expression changes). The extracted features may include facial features and non-facial features. The extracted embeddings may be output to the feature extraction module 306. A diversity measure for the top model may be trained (e.g., by transfer learning) using the extracted embeddings. The feature extraction module 306 may receive one or more of the frame scores or the extracted embeddings from the score model 305. The extracted embeddings may be utilized by the feature extraction module 306 in feature processing, which will be described below.

[0028] The feature extraction module 306 may subscribe to the camera stream 302 and receive as input an image frame 304 (e.g., 1027x768 YUV format) from the camera stream 302. The feature extraction module 306 may also receive corresponding metadata of the frame 304. The feature extraction module 306 may extract features from the frame. The extracted features may include one or more of temporal features (e.g., timestamp), facial features (e.g., facial expression, face position, number of faces), or aesthetic features (e.g., layout of objects). In an aspect, the feature extraction module 306 may receive one or more of the score or extracted embeddings (e.g., face quality score, aesthetic value score) of the frame from the score model 305.

[0029] The feature extraction module 306 performs feature processing on the frame 304 of the camera stream 302 to determine whether the frame 304 of the camera stream 302 includes interesting features (e.g., regions of interest, motion vectors, device motion, facial information, frame statistics, visual features, audio features, timestamps). The frame may be characterized as “interesting” (or not) based on the features. The feature extraction module 306 may extract one or more features from the score model 305. For example, the feature extraction module 306 may extract facial expression features from the score model 305 (e.g., a face quality score model) that are utilized to calculate a face quality score for the frame 304. The feature extraction module 306 may provide the extracted features to the feature store 308.

[0030] The feature store 308 receives and stores the extracted features from the feature extraction module 306. The extracted features may include one or more extracted embeddings (e.g., facial expression embeddings, aesthetic embeddings). The extracted features in the feature store 308 are associated with the frames 304 stored in the ring buffer 310. The feature store 308 may communicate with and send features to a feature scoring module 312. The feature scoring module 312 may perform a feature scoring process that measures one or more metrics (e.g., frame diversity, frame quality) and calculates one or more frame scores (e.g., frame diversity score, frame quality score) from a combination of the extracted embeddings for individual frames, as described below.

[0031] Also, the ring buffer 310 may subscribe to the camera stream 302. After a user presses a shutter button (e.g., shutter button 116, shutter button 118), a buffer of suggested candidate frames is kept in the ring buffer 310. The ring buffer 310 may store the last n time-stamped frames in a first in, first out (FIFO) structure. Since the ring buffer 310 has a finite capacity, the ring buffer 310 may be continually refreshed by the latest (newer) frame replacing the earliest frame in the ring buffer 310. Thus, the ring buffer 310 stores a number of captured frames ranging in time from the latest frame back to the oldest frame, and the number of frames in the ring buffer is determined by the size of the ring buffer 310.

[0032] The frame selection module 314 may perform a frame selection process on the set of frames contained in the ring buffer 310. The frame selection process may be performed serially. The frame selection module 314 represents a function that receives the frames 304 from the ring buffer 310, utilizes at least one frame score (e.g., frame quality score, frame diversity score) received from the feature scoring module 312 to determine which frames in the ring buffer 310 are unnecessary, filters the unnecessary frames (as determined by the techniques discussed herein), and provides the remaining filtered frames as output to the candidate buffer 316.

[0033] When a frame in the ring buffer 310 is deemed unnecessary may be determined by many factors. In one example, the frame selection module 314 may utilize one or more of a frame diversity score and / or a frame quality score (e.g., from the feature scoring module 312) based on the features in the feature store 308 to determine if the frame is unnecessary and should be filtered (evicted) from the ring buffer 310. In some implementations, the frame quality score and / or the frame diversity score may be calculated by the feature scoring module 312 from a combination of the extracted embeddings (features) of the individual frames. The frames in the ring buffer 310 may be sorted based on the frame quality scores in descending order, and the frames 304 may be iteratively processed to determine if the quality of a given frame is greater than a quality threshold to determine if the frame should be filtered.

[0034] The frame selection module 314 may perform a frame selection process to select at least one frame from the frame set 304 based on at least one frame score (e.g., frame quality score, frame diversity score) received from the feature scoring module 312. The frame quality score may be calculated by the feature scoring module 312 (e.g., by a frame quality scorer 402 (described below)) or by the score model 305. The frame selection module 314 may compare the calculated frame quality score to a quality threshold to determine whether the calculated frame quality score exceeds the quality threshold. If the frame selection module 314 determines that the calculated frame quality score of a frame is below a certain threshold, it may determine to evict the frame from the ring buffer 310.

[0035] The frame diversity score may be calculated by the feature scoring module 312 (e.g., by the combined frame diversity scorer 410 (described below)). If the frame selection module 314 determines that the calculated frame diversity score of the frame exceeds a quality threshold, it may generate a minimum frame diversity score for the frame. The frame selection module 314 may calculate a minimum frame diversity score for a plurality of frames in the candidate buffer 316 (e.g., all frames in the candidate buffer) based on the frame diversity score of the frame. The calculated minimum frame diversity score may be tracked by the frame selection module 314. The frame selection module 314 may further compare the minimum frame diversity score to a diversity threshold to determine whether the minimum frame diversity score is greater than the diversity threshold (e.g., exceeds the minimum diversity threshold). In response to determining that the minimum frame diversity score of the selected frame is greater than the diversity threshold, the selected frame may be stored in the candidate buffer 316 and suggested to the user 10.

[0036] The frame selection module 314 further represents a function that provides input to the candidate buffer 316 to assist in determining which frames in the candidate buffer 316 should be evicted to ensure that the candidate buffer always contains a highlight of the camera stream 302 content. Unlike the FIFO structure of the ring buffer 310, frames in the candidate buffer 316 are not necessarily dropped in insertion order, but rather according to how important the frame is to the highlight of the frames stored in the candidate buffer 316. When a frame in the candidate buffer 316 is determined to be unnecessary may be determined by a number of factors. In one example, the frame selection module 314 may utilize one or more of a frame diversity score or a frame quality score (e.g., from the feature scoring module 312) to determine whether a frame is unnecessary and should be evicted from the candidate buffer 316. The frames in the candidate buffer 316 may be sorted based on the frame quality score in descending order, and the frames 304 may be iteratively processed to determine whether the quality of a given frame is greater than a quality threshold to determine whether the frame should be filtered.

[0037] When the user 10 presses the shutter (e.g., GUI shutter button 116), a candidate buffer 316 may be created and maintained that contains candidate frames to suggest to the user. The candidate buffer 316 receives and stores remaining frames from the frame selection module 314. Because the candidate buffer 316 has a finite capacity, the frame selection module 314 may determine which frames stored in the candidate buffer 316 to evict from the candidate buffer 316 when the capacity is reached. The frame selection module 314 may compare the frame quality score (calculated by the feature scoring module 312) of a frame in the candidate buffer 316 to a quality threshold to determine whether the frame's frame quality score exceeds the quality threshold. If the frame's frame quality score is below a certain threshold, the frame may be evicted from the candidate buffer 316. If the frame quality score of a frame exceeds the quality threshold, a frame diversity scorer for the frame over multiple frames in the candidate buffer 316 may be calculated (e.g., by the feature scoring module 312) for that frame to determine a minimum frame diversity score. The minimum frame diversity score of the frame may be compared to a diversity threshold to determine whether the minimum frame diversity score is greater than the diversity threshold. In response to determining that the minimum frame diversity score is greater than the diversity threshold, the selected frame may continue to be stored in the candidate buffer 316. If the frame selection module 314 determines that the frame's frame diversity score is below a particular threshold, the frame selection module 314 may determine to evict the frame from the candidate buffer 316. By evicting less diverse and / or less quality frames from the candidate buffer 316, the frames stored in the candidate buffer 316 better represent how important the frames are to highlighting the camera stream content.

[0038] Once it is determined that a user (e.g., user 10 of FIG. 1 ) is ready to review the missed photos, the contents of the candidate buffer 316 may be analyzed and a resulting image object (e.g., an animated GIF, a stack of frames, or a collage) that highlights the camera stream 302 may be computed by a result generator module 318, and, for example, a representation of the image object may be displayed on a display (e.g., display 108).

[0039] FIG. 4 is a block diagram 400 illustrating a technique for calculating at least one frame score (e.g., frame quality score, frame diversity score) of a frame 304 utilizing a frame score generation process according to an example implementation. For example, a component of a camera manager system (camera manager 120) may be integrated with or within a camera application 114 as illustrated in FIG. 3. In the embodiment illustrated in FIG. 4, the technique is performed by a feature scoring module (e.g., feature scoring module 312 of FIG. 3). The feature scoring module is utilized to calculate at least one frame score (e.g., frame diversity score 430, frame quality score 432) of the frame 304 as described herein. The camera manager system 120 may consider multiple signals (e.g., embeddings, features) when calculating the frame score. For example, the camera manager system 120 may use a combination of a temporal diversity score 422, a facial diversity score 424, and an aesthetic diversity score 428 to compute a frame diversity score 430. The frame diversity score 430 for the frame 304 may be a weighted sum of the temporal diversity score 422 , the facial diversity score 424 , and the aesthetic diversity score 428 .

[0040] As described with respect to FIG. 3, multiple frames 304 (e.g., selected frame 304b, first frame 304a) are received, and feature processing is performed (e.g., by feature extraction module 306) to determine whether the frames 304 contain interesting features (properties) (e.g., regions of interest, motion vectors, device motion, face information, frame statistics). In feature processing, the feature extraction module (e.g., feature extraction module 306 of FIG. 3) extracts features (e.g., embeddings) from the score model and / or from the frames 304. The camera manager system 120 may store the extracted features in the feature store 308. The extracted features may be passed to various classifiers (e.g., frame quality scorer 402, temporal diversity scorer 404, facial diversity scorer 406, aesthetic diversity scorer 408) of the feature scoring module 312 for calculating at least one frame score (e.g., frame quality score 432, frame diversity score 430). One or more outputs of the classifiers (eg, frame quality score 432 , temporal diversity score 422 , facial diversity score 424 , aesthetic diversity score 428 , combined frame diversity scorer 410 ) may be utilized by the frame selection module 314 .

[0041] The feature store 308 may pass the extracted features 412 to the frame quality scorer 402, which measures a quality metric. The frame quality scorer 402 may calculate a frame quality score 432 based on the features 412. The frame quality scorer 432 may be provided as an output to the frame selection module 314 or one or more of the classifiers (e.g., face diversity scorer 406, aesthetic diversity scorer 408). The frame quality scorer 402 may generate signals such as a facial expression embedding, an aesthetic embedding, a face location embedding, a face identification embedding, and a face count embedding. Features associated with at least one face depicted in the frame (e.g., face location embedding, face identification embedding, face count embedding, facial expression embedding, facial expression change embedding, facial attribute embedding) may be provided by the frame quality scorer 402 to the face diversity scorer 406 as a face embedding 414. The frame quality scorer 402 may provide scene-related (non-facial) features depicted in the frames to the aesthetic diversity scorer 408 as aesthetic embeddings 416.

[0042] The feature store 308 may pass the time-related features 418 (e.g., timestamps) to the time diversity scorer 404. The time diversity scorer 404 may calculate a time diversity score 422 for the frames of the frame set based on one or more time-related features 418. For example, the time diversity scorer 404 may select a frame of the frame set, obtain the timestamps 418 of the selected frame 304b and the first frame 304a as features, and measure the difference between the two timestamps (timestamp difference) to generate (output) a time diversity score 422 for the pair of frames (e.g., the selected frame 304b relative to the first frame 304a). The time diversity score 422 may be provided to the combined frame diversity scorer 410. In an aspect, the first frame 304a is the frame 304 most recently received from the camera stream 302.

[0043] For example, face-related features capturing facial expressions, facial landmarks, number of faces, face locations, etc. may be determined and passed to the face diversity scorer 406. In one example, the face-related features are face features 420 passed to the face diversity scorer 406 by the feature store 308. In another example, the face-related features are face embeddings 414 passed to the face diversity scorer 406 by the frame quality scorer 402. The face diversity scorer 406 may utilize at least one of the face features 420 or face embeddings 414 in a scoring process that determines at least one facial feature difference between a pair of frames (e.g., the first frame 304a and the selected frame 304b) and calculates a face diversity score 424 for the selected frame relative to the first frame. In an embodiment, the face diversity scorer 406 takes the features of the frame pair (e.g., face features 420, face embedding 414) and generates (outputs) a face diversity score 424 for the frame pair (e.g., of the selected frame versus the first frame) using a distance metric. For example, the face diversity scorer 406 may use a distance metric to calculate the distance between the features of the selected frame 304b and the features of the first frame 304a. The face diversity scorer 406 may provide the face diversity score 424 to the combined frame diversity scorer 410. The camera manager system 120 may perform the scoring process iteratively for multiple of the frames.

[0044] Scene-related features capturing object layout, blur, camera focus, and the like may be determined and passed to the aesthetic diversity scorer 408. In one example, the scene-related features are aesthetic features 426 passed to the aesthetic diversity scorer 408 by the feature store 308. In another example, the scene-related features are aesthetic embeddings 416 passed to the aesthetic diversity scorer 408 by the frame quality scorer 402. The aesthetic diversity scorer 408 may utilize at least one of the aesthetic features 426 or the aesthetic embeddings 416 in a scoring process to determine aesthetic feature differences between a pair of frames (e.g., the first frame 304a and the selected frame 304b) and calculate an aesthetic diversity score 428. In an embodiment, the aesthetic diversity scorer 408 takes the features of the pair of frames (e.g., aesthetic features 426, aesthetic embedding 416) and uses a distance metric to calculate (output) an aesthetic diversity score 428 for the pair of frames (e.g., of the selected frame relative to the first frame). For example, a distance metric may be utilized to calculate the distance between the features of the selected frame 304b and the features of the first frame 304a to output the aesthetic diversity score 428 for the pair of frames. The distance metric used to calculate the aesthetic diversity score may be the same distance metric utilized to calculate the face diversity score or may be a different distance metric. The aesthetic diversity score measures the aesthetic feature difference between the two frames. The aesthetic diversity score 428 may be provided to the combined frame diversity scorer 410. The camera manager system 120 may repeatedly perform the scoring process on multiple of the frames. Given two image frames (e.g., the selected frame 304b and the first frame 304a), a distance metric utilized by a diversity scorer (e.g., the facial diversity scorer 406, the aesthetic diversity scorer 408) may be calculated, for example, using one or more of a Euclidean distance metric or a machine learning distance metric.The machine learning distance metric may be calculated by collecting a diversity dataset through a cloud computing platform, training a logistic regression model, and using the probability output as the frame diversity score, which is naturally scaled to [0,1].

[0045] The combined frame diversity scorer 410 may take one or more outputs of the classifiers (e.g., the temporal diversity score 422, the facial diversity score 424, the aesthetic diversity score 428) and calculate a frame diversity score 430. The frame diversity score 430 may be utilized by a frame selection module (e.g., the frame selection module 314 of FIG. 3) in a frame selection process, as described above with respect to FIG. 3. In one example, the frame diversity score is utilized by the frame selection module to determine whether to move a frame (e.g., the selected frame 304b, the first frame 304a) from the ring buffer to the candidate buffer 316. In another example, the frame diversity score is utilized by the frame selection module to select frames to keep in the ring buffer 310 and / or to select frames to evict from the ring buffer 310. In an additional example, the frame diversity score is utilized by a frame selection module to select frames stored in the candidate buffer 316 for inclusion as part of an image object generated by the result generator module 318 representing a highlight of the camera stream 302.

[0046] In the frame score generation process, a frame quality score (e.g., frame quality score 432) for the frames in the ring buffer 310 is determined, and a frame diversity score (e.g., frame diversity score 430) for the frames in the ring buffer 310 relative to the frames in the candidate buffer 316 is determined. Frames in the ring buffer 310 that are determined to have a high quality score and a high diversity score relative to the frames in the candidate buffer 316 may be added to the candidate buffer 316 and evicted from the ring buffer 310. By discarding unnecessary frames, the camera manager system 120 frees up space in the ring buffer 310, leaving only the "best" frames captured over a period of time (e.g., the past 3 seconds).

[0047] In another example, the frame diversity score 430 may be utilized by a frame selection module (e.g., frame selection module 314 of FIG. 3) in a frame selection process, as described above with respect to FIG. 3, to maintain candidate frames in the ring buffer 310 as new frames (e.g., the first frame 304a) are received by the ring buffer 310. The combined frame diversity scorer 410 may compute a frame diversity score 430 for the first frame 304a relative to all frames in the ring buffer 310 (frames already selected by the selection process). The combined frame diversity scorer 410 may utilize the frame diversity score 430 in a filtering process to filter (e.g., evict) one or more frames from the ring buffer 310. The camera manager system 120 may repeatedly perform the filtering process on multiple ones of the frames. In an exemplary filtering process repeated on multiple ones of the stored frames, a frame is selected from the stored frames, a frame quality score for the selected frame is calculated, and a drop score is assigned to the selected frame. The dropping score may be a weighted linear combination of the frame quality score 432 and the frame diversity score 430 of the selected frame. After assigning the dropping score, the camera manager system 120 may filter the frames. For example, the frame with the lowest dropping score may be determined by sorting the frames in the ring buffer 310 in descending order of dropping score and determining the frame in the ring buffer 310 with the lowest (minimum) dropping score. The camera manager system 120 may then replace the stored frame in the ring buffer 310 with the lowest dropping score with a new frame (e.g., the first frame 304a).

[0048] In the frame filtering process, if the frame selection module 314 determines that the calculated frame quality score of the selected frame in the ring buffer 310 exceeds a quality threshold, the frame selection module 314 may calculate a second frame diversity score of the selected ring buffer frame, calculated, for example, based on the frame diversity of the selected ring buffer frame iterated over a plurality of frames in the candidate buffer 316 (e.g., all frames in the candidate buffer). The second frame diversity score is used to determine a minimum (e.g., lowest) diversity score between the selected ring buffer frame and the frames in the candidate buffer 316. The frame selection module 314 may further compare the minimum diversity score to a diversity threshold to determine whether the minimum diversity score is greater than the diversity threshold (e.g., exceeds the minimum diversity threshold). In response to determining that the minimum diversity score of the selected ring buffer frame is greater than the diversity threshold, the selected ring buffer frame may be stored in the candidate buffer 316 for inclusion as part of an image object generated by the result generator module 318 representing a highlight of the camera stream 302. In one example, the camera manager system 120 may move the selected ring buffer frame from the ring buffer 310 to the candidate buffer 316. In another example, the camera manager system 120 copies the selected ring buffer frame to the candidate buffer 316 and evicts the copy of the frame from the ring buffer 310.

[0049] In an embodiment, the maximum individual score (e.g., one of the temporal diversity score 422, facial diversity score 424, or aesthetic diversity score 428) may be used by the camera manager system 120 as the frame diversity score 430. In other embodiments (not shown), the embeddings (e.g., facial expression embeddings, aesthetic embeddings, face count embeddings) generated by the frame quality scorer 402 may be wrapped as a diversity scorer frame and sent to the combined frame diversity scorer 410 to compute the frame diversity score 430.

[0050] Throughout this disclosure, examples are described in which a computing device (e.g., computing device 102) may analyze information (e.g., image data) related to a user, such as facial features extracted by feature extraction module 306 and stored in feature store 308. However, the computing device may be configured to use the information only after the computing device has received explicit permission from a user of the computing device to use the data. For example, in a situation in which computing device 102 analyzes image data for facial features to generate frame suggestions from a set of frames, individual users may be provided with an opportunity to provide input to control whether a program or function of computing device 102 may collect and utilize the data. Individual users may always control what a program may or may not do with the image data. Additionally, the collected information may be preprocessed in one or more ways before being transferred, stored, or otherwise used such that personally identifiable information is removed. For example, computing device 102 may preprocess the image data before sharing it with another device (e.g., to train a model running on another computing device) to ensure that any user-identifying or device-identifying information embedded in the data is removed. Thus, a user can control whether information about the user and the user's device is collected and, if collected, how such information may be used by the computing device and / or remote computing systems.

[0051] The entities of Figures 1, 2, 3, and 4 may be further divided, combined, used with other components, etc. In this manner, different implementations of the computing device 102 having different configurations of the camera system 104 and the camera manager 120 may be used to implement techniques and apparatus for implementing a camera manager system capable of generating frame suggestions from a frame set. The example operating environment 100 of Figure 1 and the detailed illustrations of Figures 2, 3, 4, 5, 6, and 7 are merely illustrative of some of the many possible environments and systems in which the described techniques and apparatus for implementing a camera manager system capable of generating frame suggestions from a frame set may be employed.

[0052] Exemplary Methods This section describes example methods that may operate in whole or in part separately or together. Various example methods are described, and for ease of reading, each example method is described in subsections, and the titles of these subsections are not intended to limit the interoperability of each of these methods with other methods.

[0053] FIG. 5 illustrates an exemplary method 500 performed by a computing device for generating frame proposals from a frame set. Method 500 is illustrated as a set of blocks specifying operations to be performed, but is not necessarily limited to the order or combination shown for performing the operations by each block. Furthermore, any of one or more operations may be repeated, combined, rearranged, or concatenated to provide a wide range of additional and / or alternative methods (e.g., method 600). In some of the following discussion, reference may be made to the exemplary operating environment 100 of FIG. 1, or entities or processes as detailed in other figures, but such references are for illustrative purposes only. The technology is not limited to execution by one entity or multiple entities operating on one device.

[0054] At 502, a computing device (e.g., computing device 102) receives an image data stream defining a frame set (e.g., frame 304) and a first frame (e.g., frame 304a). The image data may be received from a camera system (e.g., camera system 104) of the computing device. The frame set may include a selected frame (e.g., selected frame 304b). The computing device begins a frame score generation process at 504 to calculate frame diversity scores based on features extracted from the frames. In the frame score generation process, the computing device calculates at 506 a temporal diversity score for the frame of the frame set relative to the first frame, at 508 a facial diversity score for the frame of the frame set relative to the first frame, and at 510 an aesthetic diversity score for the frame of the frame set relative to the first frame. The computing device then calculates a diversity score for the frames of the frame set relative to the first frame based on the facial diversity score, the aesthetic diversity score, and the temporal diversity score (e.g., by combining the facial diversity score, the aesthetic diversity score, and the temporal diversity score) at 512. Using the frame diversity score, the computing device determines at 514 whether to include the first frame as part of an image object (e.g., generated by the result generator module 318) representing the proposed frame (highlight) of the image data stream.

[0055] FIG. 6 illustrates another exemplary method 600 performed by a computing device for calculating a frame score for a frame, for example, by the feature scoring module 312 of FIGS. 3 and 4. At 602, a computing device (e.g., computing device 102) receives an image data stream defining a frame set (e.g., frame 304) and a first frame (e.g., frame 304a), for example, from a camera system (e.g., camera system 104) of the computing device. At 604, the computing device extracts features from the frames and the first frame and stores the extracted features (embeddings) in a feature store (e.g., feature store 308). A feature scoring module implemented in the computing device receives the features from the feature store. The feature scoring module includes one or more classifiers (e.g., a frame quality scorer, a temporal diversity scorer, a facial diversity scorer, an aesthetic diversity scorer) that receive the features from the feature store. At 606, the frame quality scorer generates a face embedding that describes features associated with at least one face depicted in the frame. At 608, the frame quality scorer generates an aesthetic embedding that describes scene-related (non-facial) features depicted in the frame. At 610, the temporal diversity scorer generates a temporal diversity score utilizing the time-related features. At 612, the facial diversity scorer generates a facial diversity score utilizing at least one of the facial features or the facial embedding. At 614, the aesthetic diversity scorer generates an aesthetic diversity score utilizing at least one of the aesthetic features or the aesthetic embedding. At 616, a frame diversity score is generated based on the temporal diversity score, the facial diversity score, and the aesthetic diversity score (e.g., by combining the facial diversity score, the aesthetic diversity score, and the temporal diversity score). The frame diversity score is used by the computing device at 618 to select and suggest diverse frames from the set of frames.

[0056] Exemplary Computing Device FIG. 7 illustrates various components of an exemplary computing device 700 (device 700), which may be implemented as any type of client, server, and / or computing device, as described with reference to previous figures, for implementing techniques and apparatus for implementing a camera manager system capable of generating frame suggestions from a frame set.

[0057] The device 700 includes a communications device 702 that enables wired and / or wireless communication of device data 704 (e.g., received data, data being received, data to be broadcast, data packets of data). The device data 704 or other device content may include configuration settings of the device, media content stored on the device, and / or information related to a user of the device. The media content stored on the device 700 may include any type of audio, video, and / or image data. The device 700 includes one or more data inputs 706 through which any type of data, media content, and / or input may be received, including user selectable inputs (explicit or implicit), messages, music, television media content, recorded video content, and other types of audio, video, and / or image data received from any content and / or data source.

[0058] Device 700 also includes a communication interface 708, which may be implemented as any one or more of a serial and / or parallel interface, a wireless interface, any type of network interface, a modem, and any other type of communication interface. Communication interface 708 provides a connection and / or communication link between device 700 and a communication network through which other electronic, computing, and communication devices can communicate data with device 700.

[0059] The device 700 includes one or more processors 710 (e.g., either a microprocessor, a controller, etc.) that process various computer-executable instructions to control the operation of the device 700 and enable techniques for a camera manager system capable of generating frame suggestions from a frame set. Alternatively or additionally, the device 700 may be implemented in any one or combination of hardware, firmware, or fixed logic circuitry implemented in association with processing and control circuitry generally identified at 712. Although not shown, the device 700 may include a system bus or data transfer system coupling various components within the device. The system bus may include any one or combination of different bus structures including a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of a variety of bus architectures.

[0060] The device 700 also includes a computer readable medium 714 (CRM 714) including one or more memory devices that allow for persistent and / or non-transient data storage, as opposed to merely signal transmission, examples of which include random access memory (RAM), non-volatile memory (e.g., read only memory (ROM), flash memory, EPROM, EEPROM, or one or more), and disk storage devices. The disk storage may be implemented as any type of magnetic or optical storage device, such as a hard disk drive, a recordable and / or rewritable compact disc (CD), any type of digital versatile disc (DVD), or the like. The device 700 may also include a mass storage media device (storage medium) 716. The CRM 714 provides a data storage mechanism for storing the device data 704, as well as various device applications 718, and any other type of information and / or data related to the operational aspects of the device 700. For example, an operating system 720 is maintained as a computer application along with the CRM 714 and is executable on the processor(s) 710. The device applications 718 may include a device manager, e.g., any form of control application, software application, signal processing and control module, code that is native to a particular device, and a hardware abstraction layer for a particular device. The device applications 718 also include any system components, engines, or managers for implementing a camera manager system capable of generating frame suggestions from a frameset. In this example, the device applications 718 include the camera manager system 120 and the camera system 104.

[0061] These techniques and apparatus include non-transitory computer-readable storage media storing instructions that, in response to execution by one or more computer processors, perform the methods described herein, as well as systems and means for performing these methods.

[0062] As used herein, a phrase referring to "at least one" of a list of items refers to any combination of those items, including single members. By way of example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiples of the same element (e.g., aa, aa, ab, ac, ab, ac, bb, bc, cc, or any other permutation of a, b, and c).

[0063] example The following sections provide examples.

[0064] Example 1: A method (500) performed by a computing device (102), comprising receiving (502) an image data stream defining a first frame (304a) and a set of frames (304) not including the first frame, and performing (504) a frame score generation process to calculate a frame diversity score (430), the frame score generation process calculating (506) a temporal diversity score (422) for a frame of the set of frames relative to the first frame, and calculating (424) a facial diversity score (424) for a frame of the set of frames relative to the first frame. and calculating (512) a frame diversity score for the frame of the frame set for the first frame based on the facial diversity score, the aesthetic diversity score, and the temporal diversity score, the method further comprising: determining whether to include the first frame as part of an image object representing a proposed frame of the image data stream using the frame diversity score.

[0065] Example 2: The method of Example 1, wherein using the frame diversity score to determine whether to include a first frame as part of an image object representing a proposed frame of the image data stream further includes storing the set of frames (304) in a ring buffer (310) and repeatedly performing a filtering process for each stored frame, the filtering process including selecting a frame from the stored frames, calculating a frame quality score for the selected frame, assigning a drop score to the selected frame, determining the stored frame with the lowest drop score, evicting the stored frame with the lowest drop score from the ring buffer, and storing the first frame in the ring buffer.

[0066] Example 3: The method of Example 2, wherein the filtering process further includes: determining whether the calculated frame quality score of the selected ring buffer frame is greater than a quality threshold; in response to determining that the calculated frame quality score is greater than the quality threshold, calculating a minimum diversity score of the selected frame relative to the candidate frames stored in the candidate buffer; determining whether the minimum diversity score exceeds a minimum diversity threshold; and in response to determining that the minimum diversity score exceeds the minimum diversity threshold, adding the selected ring buffer frame to the candidate buffer.

[0067] Example 4: The method of Example 2 or Example 3, wherein calculating a frame quality score for the selected frame includes determining extracted facial features for the selected frame, determining extracted aesthetic features for the selected frame, and calculating the frame quality score using the extracted facial features and the extracted aesthetic features.

[0068] Example 5: The method of Example 2, Example 3, or Example 4, wherein the dropping score comprises a weighted linear combination of the frame quality score (432) and the frame diversity score (430) of the selected frame.

[0069] Example 6: A method as in any one of the preceding examples, wherein calculating a facial diversity score for a frame of a frame set relative to a first frame includes determining facial-related features (414, 420) of the first frame (304a) and iteratively performing a scoring process, the scoring process including selecting a frame (304b) from the frame set (304), determining the facial-related features of the selected frame, and utilizing a distance metric to determine a facial feature difference between the facial-related features of the selected frame and the facial-related features of the first frame.

[0070] Example 7: The method of Example 6, wherein calculating a facial diversity score for a frame of the frame set relative to the first frame further includes extracting facial features from the selected frame and the first frame, the extracted facial features representing at least one of a facial feature depicted in the frame or a facial embedding, and calculating further includes determining a facial feature difference between the selected frame and the first frame using the extracted facial features and a distance metric.

[0071] Example 8: The method of Example 7, wherein using facial features and a distance metric to determine a facial feature difference between the selected frame and the first frame includes calculating a distance between an extracted facial feature of the selected frame and an extracted facial feature of the first frame, and using the calculated distance to calculate a facial diversity score for the selected frame, the facial diversity score representing facial diversity between the selected frame and the first frame.

[0072] Example 9: A method according to any one of the preceding examples, wherein calculating an aesthetic diversity score for a frame of the frame set relative to a first frame includes determining scene-related features (416, 426) of the first frame and iteratively performing a scoring process, the scoring process including selecting a frame from the frame set, extracting scene-related features from the selected frame, and utilizing a distance metric to determine an aesthetic feature difference between the scene-related features of the selected frame and the scene-related features of the first frame.

[0073] Example 10: The method of Example 9, further comprising extracting scene-related features from the selected frame and the first frame, the extracted scene-related features representing at least one of aesthetic features or aesthetic embeddings depicted in the frames.

[0074] Example 11: The method of Example 10, wherein utilizing a distance metric to determine aesthetic feature differences between scene-related features of the selected frame and scene-related features of the first frame includes calculating a distance between the extracted scene-related features of the selected frame and the extracted scene-related features of the first frame, and utilizing the calculated distance to calculate an aesthetic diversity score for the selected frame, the aesthetic diversity score representing aesthetic diversity between the selected frame and the first frame.

[0075] Example 12: A method as described in any one of the preceding examples, wherein calculating a frame diversity score for a frame of the frame set relative to a first frame based on a facial diversity score, an aesthetic diversity score, and a temporal diversity score includes calculating a weighted sum of the facial diversity score, the aesthetic diversity score, and the temporal diversity score.

[0076] Example 13: A method as described in any one of the preceding examples, wherein calculating a temporal diversity score for a frame of the frame set relative to a first frame includes determining a timestamp difference between the selected frame and the first frame of the frame set, and generating a temporal diversity score based on the determined timestamp difference.

[0077] Example 14: The method of any one of the preceding examples, further comprising outputting a representation of the graphical object to a user for display on a display device (108).

[0078] Example 15: An apparatus comprising a camera manager system (120) configured to generate frame suggestions from a frame set (304), and a processor (11) and a memory system (112) coupled to the camera manager system (120) and configured to execute a method according to any one of Examples 1 to 14.

[0079] Example 16: A computer-readable storage medium having stored thereon computer-readable instructions that, in response to execution by one or more computer processors, cause the one or more processors to perform a method as described in any one of Examples 1-14.

[0080] conclusion Although techniques for implementing a camera manager system capable of generating frame suggestions from a frame set, and example apparatus implementations enabling such a camera manager system, have been described in feature and / or method specific language, it should be understood that the subject matter of the appended claims is not necessarily limited to the particular features or methods described. Rather, the specific features and methods are disclosed as example implementations enabling the techniques for generating frame suggestions from a frame set.

Claims

1. A method (500) performed by a computing device (102), comprising: Receiving (502) an image data stream defining a first frame (304a) and a set of frames (304) not including the first frame; and performing (504) a frame score generation process to calculate a frame diversity score (430), the frame score generation process comprising: calculating (506) a temporal diversity score (422) based on a temporal feature difference between the temporal related features of the first frame and the temporal related features of the frames of the frame set; Calculating (508) a facial diversity score (424) based on facial feature differences between facial related features of the first frame and facial related features of the frames of the frame set; Calculating (510) an aesthetic diversity score (428) based on aesthetic feature differences between scene-related features of the first frame and scene-related features of the frames of the frame set; and calculating (512) the frame diversity scores (430) for the frames of the frame set relative to the first frame based on the facial diversity score, the aesthetic diversity score, and the temporal diversity score, the method further comprising: Using the frame diversity score to determine (514) whether to include the first frame as part of an image object representing a proposed frame of the image data stream.

2. Using the frame diversity score to determine whether to include the first frame as part of an image object representing a proposed frame of the image data stream further comprises: storing said set of frames in a ring buffer (310); and repeatedly performing a filtering process for each stored frame, the filtering process comprising: selecting a frame from said stored frames; calculating a frame quality score (432) for the selected frame; assigning a drop score to the selected frame; determining the stored frame having the lowest drop score; evict from the ring buffer the stored frame having the lowest drop score; and storing the first frame in the ring buffer.

3. The filtering process further comprises: selecting a ring buffer frame stored in the ring buffer; determining whether the calculated frame quality score of the selected ring buffer frame is greater than a quality threshold; in response to determining that the calculated frame quality score is greater than the quality threshold, calculating a minimum diversity score for the selected ring buffer frame relative to candidate frames stored in a candidate buffer (316); determining whether the minimum diversity score exceeds a minimum diversity threshold; and in response to determining that the minimum diversity score exceeds the minimum diversity threshold, adding the selected ring buffer frame to the candidate buffer.

4. Calculating the frame quality scores for the selected frames includes: determining extracted facial features of the selected frames; determining extracted aesthetic features of the selected frames; and calculating the frame quality score using the extracted facial features and the extracted aesthetic features.

5. The drop score is The method of any one of claims 2 to 4, comprising a weighted linear combination of the frame quality scores (432) and the frame diversity scores (430) of the selected frames.

6. Calculating the face diversity scores for the frames of the set of frames relative to the first frame includes: determining face-related features (414, 420) of the first frame (304a); and repeatedly performing a scoring process, the scoring process comprising: selecting a frame (304b) from said set of frames (304); determining face-related features of the selected frame; and determining a facial feature difference between the face-related features of the selected frame and the face-related features of the first frame using a distance metric.

7. Calculating the face diversity scores for the frames of the set of frames relative to the first frame further includes: extracting facial features from the selected frame and the first frame, the extracted facial features representing at least one of facial features depicted in the frames or a facial embedding, and the calculating further comprises: The method of claim 6 , comprising utilizing the extracted facial features and the distance metric to determine a facial feature difference between the selected frame and the first frame.

8. Utilizing the facial features and the distance metric to determine a facial feature difference between the selected frame and the first frame includes: calculating a distance between the extracted facial features of the selected frame and the extracted facial features of the first frame; and calculating a face diversity score for the selected frame using the calculated distance, the face diversity score representing face diversity between the selected frame and the first frame.

9. Calculating the aesthetic diversity scores for the frames of the frame set relative to the first frame includes: determining scene-related features (416, 426) of the first frame; and repeatedly performing a scoring process, the scoring process comprising: selecting a frame from the set of frames; extracting scene-related features from the selected frames; and utilizing a distance metric to determine aesthetic feature differences between the scene-related features of the selected frame and the scene-related features of the first frame.

10. 10. The method of claim 9, further comprising extracting the scene-related features from the selected frame and the first frame, the extracted scene-related features representing at least one of the aesthetic features or aesthetic embeddings depicted in the frames.

11. Utilizing a distance metric to determine aesthetic feature differences between the scene-related features of the selected frame and the scene-related features of the first frame includes: calculating a distance between the extracted scene-related features of the selected frame and the extracted scene-related features of the first frame; and utilizing the calculated distance to calculate an aesthetic diversity score for the selected frame, the aesthetic diversity score representing aesthetic diversity between the selected frame and the first frame.

12. A method according to any one of claims 9 to 11, wherein the scene-related features include at least one of: a layout of objects in the corresponding frames, blur in the corresponding frames, and a focus of a camera capturing the corresponding frames.

13. Calculating the frame diversity scores for the frames of the frame set relative to the first frame based on the facial diversity score, the aesthetic diversity score, and the temporal diversity score includes: A method according to any preceding claim, comprising computing a weighted sum of the facial diversity score, the aesthetic diversity score and the temporal diversity score.

14. Calculating the temporal diversity scores for the frames of the frame set relative to the first frame includes: determining a timestamp difference between the selected frame and the first frame of the set of frames as the temporal feature difference; and generating the time diversity score based on the determined time stamp difference.

15. The method of any preceding claim, further comprising outputting a representation of the graphical object to a user for display on a display device (108).

16. The method of claim 1, further comprising displaying the image object on a display of the computing device.

17. a camera manager system (120) configured to generate frame suggestions from a set of frames (304); An apparatus comprising a processor (11) and a memory system (112) coupled to the camera manager system (120) and configured to perform the method according to any one of claims 1 to 16.

18. A computer program storing computer readable instructions which, when executed by one or more computer processors, cause the one or more computer processors to perform a method according to any one of claims 1 to 16.

Images