Camera Auto Exposure for Use in Wearable Multimedia Devices

Abstract

A system, method, device, and non-transitory computer-readable storage medium are disclosed for a wearable multimedia device and a cloud computing platform with an application ecosystem for processing data captured by the wearable multimedia device. In one embodiment, operations performed by the wearable multimedia device or the cloud computing platform include accessing information from one or more data streams, where the one or more data streams include at least one of image data or pre-set image parameters, using the information from the one or more data streams to determine exposure parameters for capturing one or more images, applying the determined exposure parameters to a camera, and controlling the camera to capture one or more images with an exposure set according to the applied exposure parameters.

Description

[Technical field]

[0001] The present disclosure relates generally to wearable multimedia devices, and in particular to using camera functionality in wearable multimedia devices. [Background technology]

[0002] Wearable devices have been developed for capturing images of a scene while being worn by a user. The wearable devices may include a camera configured to convert light with one or more sensors into one or more data values. The data values ​​may represent a scene involving one or more objects or subjects. Summary of the Invention [Means for solving the problem]

[0003] Disclosed are systems, methods, devices, and non-transitory computer-readable storage media for automatically determining exposure of an image capture device based on exposure parameters determined from acquired data. In some implementations, the image capture device is a component of a mobile device, such as a camera component in a wearable multimedia device. The acquired data can include data from a sensor onboard the multimedia device, or data acquired from a network server, or both.

[0004] In some implementations, a user uses a laser-projected virtual interface (VI), such as a high-precision laser scanner (e.g., a MEMS scanner) that can turn any surface into a VI, to interact with images generated from one or more exposures of an image capture device, such as a camera, and / or to control the operation of the image capture device. For example, a laser-projected VI can be projected onto the palm of a user's hand or other surface. A three-dimensional (3D) depth sensor (e.g., a time-of-flight (TOF) camera) can be used to detect the gestures of a user interacting with one or more VI elements projected onto a surface.

[0005] In general, a wearable multimedia device may include a projector subsystem configured to visually present information to a user in the form of projected light. For example, the projector subsystem may project light onto a surface (e.g., a surface of a user's hand, such as the palm of the user's hand) according to a particular spatial and / or temporal pattern such that the user perceives a VI with one or more user interface elements. Furthermore, the user may perform gestures to interact with the VI.

[0006] In some implementations, the wearable multimedia device can present a camera interface that allows the user to view or interact with captured images, and to indicate an intent to capture an image or initiate image capture with a predefined gesture. This VI is also referred to as the camera interface.

[0007] In some implementations, the wearable multimedia device includes an exposure engine that generates one or more exposure parameters for a component camera of the multimedia device. For example, the exposure engine can obtain data from one or more of the connected components, including an on-board sensor or a connected network server. The exposure engine can process this data to determine one or more exposure parameters. Depending on the implementation, the exposure engine can process this data using one or more of an offline trained machine learning model (also simply referred to as a model), a network-accessible trained model in a cloud computing architecture, an offline or cloud-based program algorithm, and any combination of an offline trained model, a trained model in a cloud computing architecture, and an offline or cloud-based program algorithm, or any number of offline trained models, trained models in a cloud computing architecture, and an offline or cloud-based program algorithm.

[0008] Implementations described herein can provide various technical advantages. As an example, a system including an exposure engine can reduce storage requirements of a wearable device while increasing the quality of images and enhancing a user's ability to automatically capture notable or interesting events or objects in high quality images. The system can reduce storage requirements by automatically controlling the timing of exposure to coincide with potentially notable or interesting events, such as a target object momentarily coming into view or a sudden change or movement in an exposure frame. In this manner, the system can reduce the total number of image captures. The system can reduce storage requirements by storing only images determined to be of high quality or generated high quality images and filtering out or not storing low quality images.

[0009] The system can reduce the processing power requirements of the wearable device. For example, the system can reduce the number of images captured during an exposure to only those images depicting objects or persons of interest and noteworthy or interesting events, thereby reducing processing power that would otherwise be used to process a larger set of images if the exposure was not controlled to reduce image capture during periods unlikely to contain interesting or noteworthy events. This also saves user time that would be spent with conventional image capture devices reviewing captured images and manually selecting some of them to keep and removing or deleting the rest.

[0010] In some implementations, the exposure engine provides images to a model trained to determine the quality of the images. For example, the model can be trained using training data that represents images marked by a user or algorithm as high quality, such as images that are in focus, prominently feature people, focus on objects, and display natural lighting and coloring, among others, and images marked by a user or algorithm as low quality, such as images that are out of focus, poorly focused, colored, or lit, among others. The model can also be trained based on a set of images of a user. The model can be trained to generate images with similar characteristics compared to a set of images selected by a user, a trained model, or an algorithm.

[0011] One innovative aspect of the subject matter described herein is embodied in a method that includes accessing, at a wearable device, information from one or more data streams, where the one or more data streams include at least one of image data or pre-set image parameters available to the wearable device, determining exposure parameters for capturing one or more images using the information from the one or more data streams, applying the determined exposure parameters to a camera associated with the wearable device, and controlling the camera to capture one or more images using an exposure set with the applied exposure parameters.

[0012] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs configured to perform the actions of the methods encoded on a computer storage device. A system of one or more computers may be so configured by software, firmware, hardware, or combinations thereof installed on the system that, when operated, causes the system to perform the actions. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the device to perform the actions.

[0013] Each of the foregoing and other embodiments may optionally include one or more of the following features, alone or in combination: By way of example, in some implementations, the action includes accessing information from one or more data streams in response to receiving an input to take a picture using a camera or predicting that the camera will be activated to take a picture based on a context of the wearable device determined using one or more sensors.

[0014] In some implementations, determining the exposure parameters includes adjusting pre-set image parameters based on the target features determined from the one or more data streams.

[0015] In some implementations, the pre-set image parameters are generated based on at least one of context, calendar information, or location information of the wearable device determined using one or more sensors.

[0016] In some implementations, the actions include identifying a feature in one or more data streams and detecting the feature in image data of the one or more data streams.

[0017] In some implementations, the features include at least one of a representation of a particular person or a representation of a particular object.

[0018] In some implementations, the action includes controlling a camera to adjust one or more lenses to focus on the feature. In some implementations, the action includes controlling a camera to capture the feature in one or more images.

[0019] In some implementations, the action includes controlling the camera to capture the feature in one or more additional images with an exposure set with an adjusted version of the applied exposure parameters.

[0020] In some implementations, the one or more data streams include at least one of calendar information, location information, or stored image data.

[0021] The details of the disclosed embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a block diagram of an operating environment for a wearable multimedia device and a cloud computing platform with an application ecosystem for processing multimedia data captured by the wearable multimedia device, according to one embodiment. [Diagram 2] 2 is a block diagram of a data processing system implemented by the cloud computing platform of FIG. 1 according to one embodiment. [Diagram 3] FIG. 2 is a block diagram of a data processing pipeline for processing a contextual data stream, according to one embodiment. [Figure 4]FIG. 13 is a block diagram of another data process for processing a contextual data stream for a transportation application, according to one embodiment. [Diagram 5] 3 illustrates a data object used by the data processing system of FIG. 2 according to one embodiment. [Figure 6] FIG. 1 is a flow diagram of a data pipeline process, according to one embodiment. [Figure 7] FIG. 1 is a diagram of a cloud computing platform architecture, according to one embodiment. [Figure 8] FIG. 2 is a diagram of the architecture of a wearable multimedia device according to one embodiment. [Figure 9] FIG. 1 is a system block diagram of a projector architecture, according to one embodiment. [Figure 10] FIG. 2 is a diagram of an exemplary virtual interface, according to one embodiment. [Figure 11] FIG. 1 illustrates a system for determining image exposure for a camera component of a wearable multimedia device. [Figure 12] FIG. 1 illustrates a system for determining exposure parameters and generating an image exposure. [Figure 13] FIG. 4 is a flow diagram of a process for generating an image exposure based on determined exposure parameters. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] The use of the same reference numbers in the various drawings indicates similar elements.

[0024] Exemplary Wearable Multimedia Device The features and processes described herein can be implemented on a wearable multimedia device. In one embodiment, the wearable multimedia device is a lightweight, small form factor, battery-powered device that can be attached to a user's clothing or an object using a tension clasp, an interlocking pin back, a magnet, or any other attachment mechanism. The wearable multimedia device includes a digital image capture device (e.g., a camera with 180° FOV and optical image stabilization (OIS)) that allows a user to capture multimedia data (e.g., video, audio, depth data, biometric data) of life events ("moments") and document transactions (e.g., financial transactions) spontaneously and / or continuously with minimal user interaction or device setup. The multimedia data captured by the wireless multimedia device ("context data") is uploaded to a cloud computing platform with an application ecosystem that allows the context data to be processed, edited, and formatted by one or more applications (e.g., artificial intelligence (AI) applications) into any desired presentation format (e.g., single image, image stream, video clip, audio clip, multimedia presentation, or image gallery) that can be downloaded and replayed on the wearable multimedia device and / or any other playback device. For example, the cloud computing platform can convert the video and audio data into any desired filmmaking style (e.g., documentary, lifestyle, candid, photojournalism, sports, street) specified by the user.

[0025] In one embodiment, the context data is processed by a server computer of the cloud computing platform based on the user's preferences. For example, based on the user's preferences, the image can be color graded, stabilized, and cropped to perfectly suit the moment the user wants to re-experience. The user's preferences can be stored in a user profile created by the user through an online account accessible through a website or portal, or the user's preferences can be learned over time by the platform (e.g., using machine learning). In one embodiment, the cloud computing platform is a scalable distributed computing environment. For example, the cloud computing platform can be a distributed streaming platform (e.g., Apache Kafka™) with real-time streaming data pipelines and real-time streaming applications that transform or react to the streams of data.

[0026] In one embodiment, a user can start and stop a context data capture session on the wearable multimedia device using simple touch gestures (e.g., taps or swipes), by speaking commands, or by any other input mechanism. All or a portion of the wearable multimedia device can be automatically powered down when the wearable multimedia device detects using one or more sensors (e.g., proximity sensors, optical sensors, accelerometers, gyroscopes) that it is not being worn by a user.

[0027] The contextual data may be encrypted, compressed, and stored in an online database associated with a user account using any desired encryption or compression technique. The contextual data may be stored for a specified period of time that may be set by the user. Users may be provided with opt-in mechanisms and other tools to manage their data and data privacy through a website, portal, or mobile application.

[0028] In one embodiment, the contextual data includes point cloud data to provide three-dimensional (3D) surface mapped objects that can be processed using, for example, augmented reality (AR) and virtual reality (VR) applications in the application ecosystem. The point cloud data can be generated by a depth sensor (e.g., lidar or time-of-flight (TOF)) embedded in the wearable multimedia device.

[0029] In one embodiment, the wearable multimedia device includes a Global Navigation Satellite System (GNSS) receiver (e.g., Global Positioning System (GPS)) and one or more inertial sensors (e.g., accelerometer, gyroscope) to determine the location and orientation of the user wearing the device at the time the context data was captured. In one embodiment, one or more images in the context data can be used by a location application, such as a visual odometry application, in the application ecosystem to determine the user's position and orientation.

[0030] In one embodiment, the wearable multimedia device may also include one or more environmental sensors, including but not limited to ambient light sensors, magnetometers, pressure sensors, voice activity detectors, etc. This sensor data may be included in the context data to enhance the content presentation with additional information that may be used to capture the moment.

[0031] In one embodiment, the wearable multimedia device may include one or more biometric sensors, such as a heart rate sensor, a fingerprint scanner, etc. This sensor data may be included in the context data to document a transaction or to indicate the user's emotional state during a moment (e.g., an elevated heart rate may indicate excitement or anxiety).

[0032] In one embodiment, the wearable multimedia device includes a headphone jack for connecting a headset or earphones, and one or more microphones for receiving voice commands and capturing ambient audio. In an alternative embodiment, the wearable multimedia device includes short-range communication technologies, including but not limited to Bluetooth, IEEE 802.15.4 (ZigBee™), and Near Field Communication (NFC). The short-range communication technologies can be used to wirelessly connect to a wireless headset or earphones in addition to or instead of the headphone jack, and / or the short-range communication technologies can wirelessly connect to any other external devices (e.g., computers, printers, projectors, televisions, and other wearable devices).

[0033] In one embodiment, the wearable multimedia device includes a wireless transceiver and communication protocol stacks for various communication technologies, including Wi-Fi, 3G, 4G, and 5G communication technologies. In one embodiment, the headset or earphones also include sensors (e.g., biometric sensors, inertial sensors) that provide information about the direction the user is facing, such as to provide commands using head gestures or to play spatial audio. In one embodiment, the camera direction can be controlled by head gestures so that the camera's field of view follows the user's field of view. In one embodiment, the wearable multimedia device can be embedded or attached to the user's glasses.

[0034] In one embodiment, the wearable multimedia device includes a projector (e.g., a laser projector) or other digital projection technology (e.g., Liquid Crystal on Silicon (LCoS or LCOS) technology, Digital Light Processing (DLP) technology, or Liquid Crystal Display (LCD) technology) or can be wired or wirelessly coupled to an external projector, allowing a user to replay moments on a surface such as a wall or tabletop, or on the surface of the user's hand (e.g., the palm of the user's hand). In another embodiment, the wearable multimedia device includes an output port that can be connected to a projector or other output device.

[0035] In one embodiment, the wearable multimedia capturing device includes a touch surface that responds to touch gestures (e.g., tap gestures, multi-tap gestures, or swipe gestures). The wearable multimedia device may include a small display for presenting information and one or more light indicators for indicating on / off status, power status, or any other desired status.

[0036] In one embodiment, the cloud computing platform can be driven by context-based gestures (e.g., air gestures) combined with speech queries, such as a user pointing at an object in their environment and saying "What's that building?" The cloud computing platform uses the air gestures to narrow the camera's field of view to isolate the building. One or more images of the building are captured, optionally cropped (e.g., to protect privacy), and sent to the cloud computing platform, where an image recognition application can run the image query and store or return the results to the user. Air and touch gestures can also be implemented on a projected temporary display, for example in response to user interface elements projected onto a surface.

[0037] In one embodiment, the contextual data can be encrypted on the device and on a cloud computing platform so that only the user or any authorized viewer can re-experience the moment on a connected screen (e.g., smartphone, computer, television, etc.) or as a projection on a surface. An exemplary architecture of a wearable multimedia device is described with reference to FIG.

[0038] In addition to personal life events, wearable multimedia devices also facilitate the capture of financial transactions currently handled by smartphones. Capture of everyday transactions (e.g., commercial transactions, microtransactions) becomes easier, faster, and smoother by using the sight assisted contextual awareness provided by wearable multimedia devices. For example, when a user is involved in a financial transaction (e.g., making a purchase), the wearable multimedia device generates data memorializing the financial transaction, including date, time, amount, digital images or video of the parties involved, audio (e.g., the user's narration describing the transaction), and environmental data (e.g., location data). The data can be included in a multimedia data stream sent to a cloud computing platform where it can be stored online and / or processed by one or more financial applications (e.g., financial management, accounting, budgeting, tax reporting, inventory, etc.).

[0039] In one embodiment, the cloud computing platform provides a graphical user interface on a website or portal that enables various third-party application developers to upload, update, and manage their applications within the application ecosystem. Some exemplary applications can include, but are not limited to, personal live broadcasting (e.g., Instagram™ Life, Snapchat™), elderly monitoring (e.g., to make sure a loved one has taken their medication), memory recall (e.g., showing your child's soccer game last week), and personal guidance (e.g., an AI-enabled personal guidance that knows the user's location and guides the user to perform an action).

[0040] In one embodiment, the wearable multimedia device includes one or more microphones and a headset. In some embodiments, the headset wire includes the microphone. In one embodiment, a digital assistant is implemented on the wearable multimedia device that responds to user queries, requests, and commands. For example, a wearable multimedia device worn by a parent captures instantaneous context data of a child's soccer game, particularly the "moment" when the child scores a goal. The user can request (e.g., using a speech command) that the platform create a video clip of the goal and store it in his or her user account. Without any further action by the user, the cloud computing platform identifies the appropriate portion of the instantaneous context data when the goal is scored (e.g., using facial recognition, visual cues, or audio cues), compiles the instantaneous context data into a video clip, and stores the video clip in a database associated with the user account.

[0041] In one embodiment, the wearable multimedia device may include photovoltaic surface technology to preserve battery life, and inductive charging circuitry (e.g., Qi) to enable inductive charging on a charging mat and wireless over-the-air (OTA) charging.

[0042] In one embodiment, the wearable multimedia device is configured to magnetically couple or engage with a rechargeable portable battery pack. The portable battery pack includes an engagement surface having a permanent magnet (e.g., a north pole) disposed thereon, and the wearable multimedia device has a corresponding engagement surface having a permanent magnet (e.g., a south pole) disposed thereon. Any number of permanent magnets having any desired shape or size can be arranged in any desired pattern on the engagement surfaces.

[0043] The permanent magnets hold the portable battery pack and the wearable multimedia device together in an engaged configuration with clothing (e.g., the user's shirt) between them. In one embodiment, the portable battery pack and the wearable multimedia device have the same engagement surface dimensions and therefore no overhanging portions when in the engaged configuration. The user magnetically secures the wearable multimedia device to their clothing by placing the portable battery pack under their clothing and placing the wearable multimedia device on top of the portable battery pack outside their clothing, thereby causing the permanent magnets to attract each other through the clothing.

[0044] In one embodiment, the portable battery pack has a built-in wireless power transmitter that is used to wirelessly power the wearable multimedia device while in the engaged configuration using principles of resonant inductive coupling, hi one embodiment, the wearable multimedia device includes a built-in wireless power receiver that is used to receive power from the portable battery pack while in the engaged configuration.

[0045] System Overview 1 is a block diagram of an operating environment for a wearable multimedia device and a cloud computing platform with an application ecosystem for processing multimedia data captured by the wearable multimedia device, according to one embodiment. The operating environment 100 includes a wearable multimedia device 101, a cloud computing platform 102, a network 103, an application ("app") developer 104, and a third-party platform 105. The cloud computing platform 102 is coupled to one or more databases 106 for storing contextual data uploaded by the wearable multimedia device 101.

[0046] As previously described, the wearable multimedia device 101 is a lightweight, small form factor, battery-powered device that can be attached to a user's clothing or an object using a tension clasp, interlocking pinbacks, magnets, or any other attachment mechanism. The wearable multimedia device 101 includes a digital image capture device (e.g., a camera with 180° FOV and OIS) that allows a user to spontaneously capture "in-the-moment" multimedia data (e.g., video, audio, depth data) and document daily transactions (e.g., financial transactions) with minimal user interaction or device setup. The contextual data captured by the wireless multimedia device 101 is uploaded to the cloud computing platform 102. The cloud computing platform 102 includes an application ecosystem that allows the contextual data to be processed, edited, and formatted by one or more server-side applications into any desired presentation format (e.g., single image, image stream, video clip, audio clip, multimedia presentation, image gallery) that can be downloaded and replayed on the wearable multimedia device and / or other playback devices.

[0047] As an example, at a child's birthday party, a parent can clip the wearable multimedia device to their clothing (or attach the device to a necklace or chain and wear it around their neck) with the camera lens facing their line of sight. The camera includes a 180° FOV, which allows the camera to capture almost everything the user is currently looking at. The user can simply tap the surface of the device, or press a button, or speak a command to start recording. No additional setup is required. A multimedia data stream (e.g., video with audio) is recorded capturing special moments of the birthday (e.g., blowing out the candles). This "context data" is sent in real time to the cloud computing platform 102 over a wireless network (e.g., Wi-Fi, cellular). In one embodiment, the context data is stored on the wearable multimedia device so that it can be uploaded at a later time. In another embodiment, the user can transfer the context data to another device (e.g., a personal computer hard drive, a smartphone, a tablet computer, a thumb drive) and later use an application to upload the context data to the cloud computing platform 102.

[0048] In one embodiment, the contextual data is processed by one or more applications of an application ecosystem hosted and managed by the cloud computing platform 102. The applications can be accessed through their respective application programming interfaces (APIs). A custom distributed streaming pipeline is created by the cloud computing platform 102 to process the contextual data based on one or more of data type, data volume, data quality, user preferences, templates, and / or any other information to generate a desired presentation based on the user preferences. In one embodiment, machine learning techniques can be used to automatically select appropriate applications for inclusion in the data processing pipeline, with or without user preferences. For example, historical user context data stored in a database (e.g., a NoSQL database) can be used to determine the user preferences for data processing using any suitable machine learning techniques (e.g., deep learning neural networks or convolutional neural networks).

[0049] In one embodiment, the application ecosystem can include a third-party platform 105 that processes the context data. A secure session is set up between the cloud computing platform 102 and the third-party platform 105 to send / receive the context data. This design allows the third-party app provider to control access to their application and provide updates. In another embodiment, the application runs on the cloud computing platform 102's servers and updates are sent to the cloud computing platform 102. In the latter embodiment, the app developer 104 can use APIs provided by the cloud computing platform 102 to upload and update applications to be included in the application ecosystem.

[0050] Exemplary Data Processing System Figure 2 is a block diagram of a data processing system implemented by the wearable multimedia device and cloud computing platform of Figure 1 according to one embodiment. The data processing system 200 includes a recorder 201, a video buffer 202, an audio buffer 203, a photo buffer 204, a capture server 205, a data store 206, a video processor 207, an audio processor 208, a photo processor 209, and a third party processor 210.

[0051] A recorder 201 (e.g., a software application) running on the wearable multimedia device records video, audio, and photo data ("context data") captured by the camera and audio subsystems and stores them in buffers 202, 203, 204, respectively. This context data is then sent (e.g., using wireless OTA technology) to an ingestion server 205 of the cloud computing platform 102. In one embodiment, the data can be sent in separate data streams, each with a unique stream identifier (streamid). A stream is a separate piece of data that can include the following exemplary attributes: location (e.g., latitude, longitude), user, audio data, a video stream of variable duration, and N photos. A stream can have a duration from 1 to MAXSTREAM_LEN seconds, where in this example MAXSTREAM_LEN=20 seconds.

[0052] The ingestion server 205 ingests the streams and creates stream records in the data store 206 to store the results of the processors 207-209. In one embodiment, the audio stream is processed first and used to determine other streams that are required. The ingestion server 205 sends the streams to the appropriate processors 207-209 based on the streamid. For example, the video stream is sent to the video processor 207, the audio stream is sent to the audio processor 208, and the photo stream is sent to the photo processor 209. In one embodiment, at least a portion of the data (e.g., image data) collected from the wearable multimedia device is processed into metadata and encrypted so that it can be further processed by a given application and sent back to the wearable multimedia device or other devices.

[0053] The processors 207-209 may execute proprietary or third-party applications as previously described. For example, the video processor 207 may be a video processing server that sends raw video data stored in the video buffer 202 to a set of one or more image processing / editing applications 211, 212 based on user preferences or other information. The processor 207 sends requests to the applications 211, 212 and returns results to the capture server 205. In one embodiment, the third-party processor 210 may process one or more of the streams using its own processor and application 217. In another example, the audio processor 208 may be an audio processing server that sends speech data stored in the audio buffer 203 to speech-to-text applications 213, 214. In another example, the photo processor 209 may be an image processing server that sends image data stored in the photo buffer 204 to image processing applications 215, 216.

[0054] Exemplary Scene Identification Application 3 is a block diagram of a data processing pipeline for processing a contextual data stream, according to one embodiment. In this embodiment, the data processing pipeline 300 is created and configured to determine what a user is watching based on contextual data captured by a wearable multimedia device worn by the user. The ingestion server 301 receives an audio stream (e.g., including a user's narration) from the audio buffer 203 of the wearable multimedia device and sends the audio stream to an audio processor 305. The audio processor 305 sends the audio stream to an app 306, which performs speech-to-text conversion and returns parsed text to the audio processor 305. The audio processor 305 returns the parsed text to the ingestion server 301.

[0055] The video processor 302 receives the parsed text from the capture server 301 and sends a request to the video processing app 307. The video processing app 307 identifies objects in the video scene and labels the objects using the parsed text. The video processing app 307 sends a response describing the scene (e.g., labeled objects) to the video processor 302. The video processor then forwards the response to the capture server 301. The capture server 301 sends the response to the data merge process 308, which merges the response with the user's location, orientation, and map data. The data merge process 308 returns the response with the scene description to the recorder 304 on the wearable multimedia device. For example, the response may include text describing the scene as a child's birthday party, including a map location and a description of the objects in the scene (e.g., identifying people in the scene). The recorder 304 associates the scene description with the multimedia data stored on the wearable multimedia device (eg, using the streamid). When the user recalls the data, the data is enriched with the scene description.

[0056] In one embodiment, the data merge process 308 can use more than just location and map data. There may be concepts of ontologies. For example, facial features of a user's father captured in an image may be recognized by a cloud computing platform and returned as "Dad" rather than the user's name, and an address such as "555 Main Street, San Francisco, CA" may be returned as "Home." Ontologies can be specific to a user and can grow and learn from the user's input.

[0057] Example Transportation Application FIG. 4 is another data processing block diagram for processing a contextual data stream for a transportation application, according to one embodiment. In this embodiment, a data processing pipeline 400 is created to call a transportation company (e.g., Uber®, Lyft®) for a ride home. Contextual data from the wearable multimedia device is received by an ingestion server 401, and an audio stream from the audio buffer 203 is sent to an audio processor 405. The audio processor 405 sends the audio stream to an app 406, which converts the speech to text. The parsed text is returned to the audio processor 405, which returns the parsed text (e.g., a user speech request for transportation) to the ingestion server 401. The processed text is sent to a third party processor 402. The third party processor 402 sends the user location and a token to a third party application 407 (e.g., an Uber® application or a Lyft™® application). In one embodiment, the token is an API and authorization token used to mediate requests on behalf of the user. The application 407 returns a response data structure to the third party processor 402, which forwards the response data structure to the ingestion server 401. The ingestion server 401 checks the ride arrival status (e.g., ETA) in the response data structure and sets up a callback to the user in the user callback queue 408. The ingestion server 401 returns a response with a vehicle description to the recorder 404, which the digital assistant can speak to the user through a loudspeaker on the wearable multimedia device or through the user's headphones or earphones via a wired or wireless connection.

[0058] FIG. 5 illustrates data objects used by the data processing system of FIG. 2 according to one embodiment. The data objects are part of a software component infrastructure instantiated on a cloud computing platform. The “Streams” object includes the following data: streamid, deviceid, start, end, lat, lon, attributes, and entities. “streamid” identifies the stream (e.g., video, audio, photo), “deviceid” identifies the wearable multimedia device (e.g., mobile device ID), “start” is the start time of the context data stream, “end” is the end time of the context data stream, “lat” is the latitude of the wearable multimedia device, “lon” is the longitude of the wearable multimedia device, “attributes” include, e.g., birth date, facial point, skin tone, audio characteristics, address, phone number, etc., and “entities” constitute the ontology. For example, the name “John Do” is mapped to “Dad” or “Brother” depending on the user.

[0059] The "Users" object includes the following data: userid, deviceid, email, fname, and lname. userid identifies the user with a unique identifier, deviceid identifies the wearable device with a unique identifier, email is the user's registered email address, fname is the user's first name, and lname is the user's last name. The "Userdevices" object includes the following data: userid and deviceid. The "Devices" object includes the following data: deviceid, started, state, modified, and created. In one embodiment, deviceid is a unique identifier for the device (different from, for example, a MAC address). started is when the device was first started. state is on / off / sleep. modified is the last update date reflecting the last state change or operating system (OS) change. created is when the device was turned on for the first time.

[0060] The "ProcessingResults" object contains the following data: streamid, ai, result, callback, duration, and accuracy. In one embodiment, streamid is each user stream as a universally unique identifier (UUID). For example, a stream that started between 8:00AM and 10:00AM will have id:15h158dhb4, and a stream that starts between 10:15AM and 10:18AM will have the UUID that was contacted for this stream. ai is the identifier of the platform application that was contacted for this stream. result is the data sent from the platform application. callback is the callback that was used (the callback is tracked in case the version changes and thus the platform needs to replay the request). accuracy is a score on how accurate the result set is. In one embodiment, the processing results can be used for multiple tasks, such as 1) notifying the merge server of the set of results, 2) determining the fastest ai so that the user experience can be improved, and 3) determining the most accurate ai. Depending on the use case, speed may be preferred over accuracy, or vice versa.

[0061] An "Entities" object contains the following data: entityID, userID, entityName, entityType, and entityAttribute. entityID is the UUID of the entity, and if an entity has multiple entries, the entityID refers to one of the entities. For example, "Barack Obama" would have an entityID of 144, which can be linked in an association table to POTUS44, or "Barack Hussein Obama", or "President Obama". userID identifies the user who created the entity record. entityName is what the userID calls the entity. For example, the entityName for Malia Obama's entityID 144 could be "Dad" or "Papa". entityType is a person, place, or thing. entityAttributes are a number of attributes about the entity, which are specific to the userID's understanding of that entity. This allows entities to map one to the other, so, for example, when Malia utters a speech query like "Can you see my dad?", the cloud computing platform can translate that query to Barack Hussein Obama and use that when brokering requests to third parties or looking up information within the system.

[0062] Example Process 6 is a flow diagram of a data pipeline process according to one embodiment. The process 600 can be implemented using the wearable multimedia device 101 and cloud computing platform 102 described with reference to FIGS. 1-5.

[0063] Process 600 can begin with receiving contextual data from a wearable multimedia device (601). For example, the contextual data can include video, audio, and still images captured by the camera and audio subsystems of the wearable multimedia device.

[0064] The process 600 can continue by creating (e.g., instantiating) 602 a data processing pipeline with applications based on the context data and the user's requirements / preferences. For example, based on the user's requirements or preferences and also based on the data type (e.g., audio, video, photo), one or more applications can be logically connected to form a data processing pipeline for processing the context data into a presentation that is played on the wearable multimedia device or another device.

[0065] The process 600 can continue by processing 603 the contextual data in a data processing pipeline. For example, speech from a user's narration during a moment or transaction can be converted to text, which is then used to label objects in a video clip.

[0066] The process 600 may continue by sending (604) the output of the data processing pipeline to a wearable multimedia device and / or other playback device.

[0067] Exemplary Architecture of a Cloud Computing Platform FIG. 7 is an exemplary architecture 700 of the cloud computing platform 102 described with reference to FIGS. 1-6, according to one embodiment. Other architectures are possible, including architectures with more or fewer components. In some implementations, the architecture 700 includes one or more processors 702 (e.g., a dual-core Intel® Xeon® processor), one or more network interfaces 706, one or more storage devices 704 (e.g., hard disks, optical disks, flash memory), and one or more computer-readable media 708 (e.g., hard disks, optical disks, flash memory, etc.). These components can communicate and exchange data via one or more communication channels 710 (e.g., a bus), which can utilize various hardware and software to facilitate the transfer of data and control signals between the components.

[0068] The term "computer-readable medium" refers to any medium that participates in providing instructions to the processor 702 for execution, including, but not limited to, non-volatile media (e.g., optical or magnetic disks), volatile media (e.g., memory), and transmission media. Transmission media include, but are not limited to, coaxial cables, copper wire, and fiber optics.

[0069] The computer readable medium 708 may further include an operating system 712 (eg, Mac OS Server, Windows NT Server, Linux Server), a network communication module 714 , data processing instructions 716 , and interface instructions 718 .

[0070] The operating system 712 may be multi-user, multi-processing, multi-tasking, multi-threading, real-time, etc. The operating system 712 performs basic tasks including, but not limited to, recognizing input from and providing output to the devices 702, 704, 706, and 708, keeping track of and managing files and directories on the computer-readable medium 708 (e.g., memory or storage device), controlling peripheral devices, and managing traffic on one or more communication channels 710. The network communication module 714 includes various components for establishing and maintaining network connections (e.g., software for implementing communication protocols such as TCP / IP, HTTP, etc.), and various components for creating a distributed streaming platform using, for example, Apache Kafka™. The data processing instructions 716 include server-side or back-end software for implementing the server-side operations described with reference to Figures 1-6. The interface instructions 718 include software for implementing a web server and / or portal for sending data to and receiving data from the wearable multimedia device 101, the third party application developer 104, and the third party platform 105 described with reference to FIG. 1.

[0071] The architecture 700 can be included in any computing device, including one or more server computers in a local or distributed network, each having one or more processing cores. The architecture 700 can be implemented in a parallel processing infrastructure or a peer-to-peer infrastructure, or on a single device with one or more processors. The software can include multiple software components or can be a single body of code.

[0072] Exemplary Architecture of a Wearable Multimedia Device Figure 8 is a block diagram of an example architecture 800 of a wearable multimedia device implementing the features and processes described with reference to Figures 1-6. The architecture 800 may include a memory interface 802, a data processor, image processor, or central processing unit 804, and a peripheral interface 806. The memory interface 802, the processor 804, or the peripheral interface 806 may be separate components or may be integrated in one or more integrated circuits. One or more communication buses or signal lines may couple the various components.

[0073] Sensors, devices, and subsystems can be coupled to the peripherals interface 806 to facilitate multiple functions. For example, a motion sensor 810, a biometric sensor 812, and a depth sensor 814 can be coupled to the peripherals interface 806 to facilitate motion, orientation, biometric, and depth detection functions. In some implementations, motion sensors 810 (e.g., accelerometers, rate gyroscopes) can be utilized to detect movement and orientation of the wearable multimedia device.

[0074] Other sensors, such as environmental sensors (e.g., temperature sensors, barometers, ambient light), can also be connected to the peripherals interface 806 to facilitate environmental sensing functions. For example, biometric sensors can detect fingerprints, facial recognition, heart rate, and other fitness parameters. In one embodiment, a haptic motor (not shown) can be coupled to the peripherals interface and can provide vibration patterns as haptic feedback to the user.

[0075] To enable georeferencing, a location processor 815 (e.g., a GNSS receiver chip) may be connected to the peripherals interface 806. An electronic magnetometer 816 (e.g., an integrated circuit chip) may also be connected to the peripherals interface 806 to provide data that may be used to determine the direction of magnetic north. Thus, the electronic magnetometer 816 may be used by an electronic compass application.

[0076] A camera subsystem 820 and an optical sensor 822, such as a charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) optical sensor, may be utilized to facilitate camera functions such as recording of photographs and video clips. In one embodiment, the camera has a 180° FOV and OIS. The depth sensor may include an infrared emitter that projects a known pattern of dots onto the object / subject. The dots are then photographed by a dedicated infrared camera and analyzed to determine depth data. In one embodiment, a time-of-flight (TOF) camera may be used to determine distance based on the known speed of light and measuring the time of flight of the light signal between the camera and the object / subject for each point in the image.

[0077] The communication functions may be facilitated through one or more communication subsystems 824. The communication subsystem 824 may include one or more wireless communication subsystems. The wireless communication subsystem 824 may include radio frequency receivers and transmitters and / or optical (e.g., infrared) receivers and transmitters. The wired communication system may include a port device, such as a Universal Serial Bus (USB) port or some other wired port connection, that may be used to establish a wired connection to other computing devices, such as other communication devices, network access devices, personal computers, printers, display screens, or other processing devices capable of receiving or transmitting data (e.g., projectors).

[0078] The specific design and implementation of the communications subsystem 824 may depend on the communications network or medium over which the device is intended to operate. For example, the device may include wireless communications subsystems designed to operate over Global System for Mobile Communications (GSM) networks, GPRS networks, Enhanced Data GSM Environment (EDGE) networks, IEEE 802.xx communications networks (e.g., Wi-Fi, WiMax, ZigBee™), 3G networks, 4G networks, 4G LTE networks, Code Division Multiple Access (CDMA) networks, Near Field Communication (NFC), Wi-Fi Direct, and Bluetooth™ networks. The wireless communications subsystem 824 may include a hosting protocol such that the device may be configured as a base station for other wireless devices. As another example, the communications subsystem may enable the device to synchronize with a host device using one or more protocols or communications technologies, such as, for example, the TCP / IP protocol, the HTTP protocol, the UDP protocol, the ICMP protocol, the POP protocol, the FTP protocol, the IMAP protocol, the DCOM protocol, the DDE protocol, the SOAP protocol, HTTP Live Streaming, MPEG Dash, and any other known communications protocol or technology.

[0079] An audio subsystem 826 may be coupled to a speaker 828 and one or more microphones 830 to facilitate voice-enabled functions such as voice recognition, voice duplication, digital recording, telephony, and beamforming.

[0080] The I / O subsystem 840 can include a touch controller 842 and / or another input controller 844. The touch controller 842 can be coupled to a touch surface 846. The touch surface 846 and touch controller 842 can detect contact and movement or their interruption using any of a number of touch sensitivity technologies including, for example, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements to determine one or more points of contact with the touch surface 846. In one implementation, the touch surface 846 can display virtual or soft buttons, which a user can use as an input / output device.

[0081] Other input controllers 844 may be coupled to other input / control devices 848, such as one or more buttons, rocker switches, thumb wheels, infrared ports, USB ports, and / or a pointer device such as a stylus. The one or more buttons (not shown) may include up / down buttons for adjusting the volume of the speaker 828 and / or microphone 830.

[0082] Additionally, a projector subsystem 832 can be connected to the peripherals interface 806 to visually present information to the user in the form of projected light. For example, the projector subsystem 832 can project light onto a surface according to a particular spatial and / or temporal pattern such that the user perceives text, images, videos, colors, patterns, and / or any other graphical information on the surface. In some implementations, the projector subsystem 832 can project light onto a surface of the user's body, such as the user's hand or palm. In some implementations, the projector subsystem 832 can project light onto a surface other than the user's body, such as a wall, table, desk, or any other object. The projector subsystem 832 is described in more detail with reference to FIG. 9.

[0083] In some implementations, the projector subsystem 832 projects light onto a surface to provide interactive VIs to a user. For example, the projector subsystem 832 can project light onto a surface so that the user perceives one or more interactive user interface elements (e.g., selectable buttons, dials, switches, boxes, images, videos, text, icons, etc.). Furthermore, the user can interact with the VIs by performing one or more gestures on the VIs and user interface elements. For example, the user can perform a pointing gesture, a tap gesture, a swipe gesture, a wave gesture, or any other gesture using his or her hand and / or fingers. The wearable multimedia device can detect the performed gestures using one or more sensors (e.g., the camera / video subsystem 820, the environmental sensor 817, the depth sensor 814, etc.), identify one or more commands associated with the gestures, and execute the identified commands (e.g., using the processor 804). Exemplary VIs are described in more detail below.

[0084] In some implementations, device 800 plays recorded audio and / or video files (including spatial audio) to a user, such as MP3 files, AAC files, spatial audio files, and MPEG video files. In some implementations, device 800 can include MP3 player functionality and can include pin connectors or other ports for interfacing with other devices. Other input / output and control devices can also be used. In one embodiment, device 800 can include an audio processing unit for streaming audio to an accessory device via a direct or indirect communication link.

[0085] The memory interface 802 can be coupled to a memory 850. The memory 850 can include high-speed random access memory or non-volatile memory such as one or more magnetic disk storage devices, one or more optical storage devices, or flash memory (e.g., NAND, NOR). The memory 850 can store an operating system 852, such as Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks. The operating system 852 can include instructions for handling basic system services and for performing hardware-dependent tasks. In some implementations, the operating system 852 can include a kernel (e.g., a UNIX kernel).

[0086] The memory 850 may also store communications instructions 854 for facilitating communications with one or more additional devices, one or more computers or servers, including peer-to-peer communications with wireless accessory devices, as described with reference to Figures 1-6. The communications instructions 854 may also be used to select an operating mode or communications medium for use by the device based on the geographic location of the device.

[0087] Memory 850 may include sensor processing instructions 858 for facilitating sensor-related processing and functions, and recorder instructions 860 for facilitating recording functions, as described with reference to Figures 1-6. Other instructions may include GNSS / navigation instructions for facilitating GNSS and navigation-related processes, camera instructions for facilitating camera-related processes, and user interface instructions for facilitating user interface processing, including touch models for interpreting touch input.

[0088] Each of the above-identified instructions and applications may correspond to a set of instructions for performing one or more functions described above. These instructions need not be implemented as separate software programs, procedures, or modules. Memory 850 may include additional instructions or fewer instructions. Furthermore, various functions of the device may be implemented in hardware and / or software, including in one or more signal processing circuits and / or application specific integrated circuits (ASICs).

[0089] 9 is a system block diagram of a projector subsystem 832, according to one embodiment. The projector subsystem 832 scans pixels in two dimensions, images a 2D array of pixels, or mixes imaging and scanning. Scanning projectors directly utilize the narrow divergence and two-dimensional (2D) scanning of a laser beam to "paint" an image pixel by pixel. In some embodiments, separate scanners are used for the horizontal and vertical scan directions. In other embodiments, a single two-axis scanner is used. The specific beam trajectory also varies depending on the type of scanner used.

[0090] In the illustrated example, the projector subsystem 832 is a scanning picoprojector that includes a controller 901, a battery 902, a power management chip (PMIC) 903, a solid-state laser 904, an XY scanner 905, a driver 906, a memory 907, a digital-to-analog converter (DAC) 908, and an analog-to-digital converter (ADC) 909.

[0091] The controller 901 provides control signals to the XY scanner 905. In response to the control signals, the XY scanner 905 uses movable mirrors to steer the laser beam generated by the solid-state laser 904 in two dimensions. The XY scanner 905 includes one or more microelectromechanical (MEMS) micromirrors with tilt angles controllable in one or two dimensions. The driver 906 includes power amplifiers and other electronic circuits (e.g., filters, switches) to provide control signals (e.g., voltage or current) to the XY scanner 905. The memory 907 stores various data used by the projector, including laser patterns for projected text and images. The DAC 908 and ADC 909 perform data conversion between the digital and analog domains. The PMIC 903 manages the power and duty cycle of the solid-state laser 904, including turning it on and off and adjusting the amount of power provided to the solid-state laser 904. The solid-state laser 904 can be, for example, a vertical cavity surface emitting laser (VCSEL).

[0092] In one embodiment, the controller 901 uses image data from the camera / video subsystem 820 and / or depth data from the depth sensor 814 to recognize and track the position of the user's hands and / or fingers on the laser projection so that user input is received by the wearable multimedia device 101 using the laser projection as an input interface.

[0093] In another embodiment, the projector subsystem 832 uses a vector graphic projection display and low power fixed MEMS micromirrors to conserve power. The projector subsystem 832 includes a depth sensor so that the projected area can be masked as needed to avoid projecting onto a finger / hand interacting with the laser projected image. In one embodiment, the depth sensor can also track gestures to control input on another device (e.g., swiping an image on a TV screen, interacting with a computer, a smart speaker, etc.).

[0094] In other embodiments, liquid crystal on silicon (LCoS or LCOS) digital projection technology, digital light processing (DLP) digital projection technology, or liquid crystal display (LCD) digital projection technology can be used instead of the picoprojector.

[0095] Example Virtual Interface As described above, the wearable multimedia device 101 can include a projector subsystem 832 configured to visually present information to a user in the form of projected light. For example, the projector subsystem 832 can project light onto a surface (e.g., the surface of a user's hand, such as the palm of the user's hand) according to a particular spatial and / or temporal pattern such that the user perceives a VI with one or more user interface elements. In some implementations, the VI and / or user interface elements can include any combination of text, images, videos, colors, patterns, shapes, lines, or any other graphical information.

[0096] Additionally, the user can perform gestures to interact with the VI. By way of example, the user can perform one or more gestures directed at one or more of the user interface elements. By way of example, the user can point at a user interface element, touch or tap a user interface element (e.g., once or multiple times in sequence) with his / her finger, perform a swiping motion along a user interface element with his / her finger, wave at a user interface element with his / her hand, hover over a user interface element, or perform any other gesture with a hand or finger. The wearable multimedia device 101 can detect the performed gestures using one or more sensors (e.g., the camera / video subsystem 820, the environmental sensor 817, the depth sensor 814, etc.), identify one or more commands associated with the gestures, and execute the identified commands (e.g., using the processor 804).

[0097] At least some of the user interface elements and / or commands can be used to control the operation of the wearable multimedia device 101. For example, at least some of the user interface elements and / or commands can be used to perform or control the generation of video and / or audio content, the viewing of content, the editing of content, the storage and transmission of data, and / or any other operation described herein. In some implementations, the wearable multimedia device 101 can present a VI that allows a user to view images or control a camera device, and to interact with a presented image by marking the presented image as a favorite or sharing the presented image with one or more other users. This VI can also be referred to as a camera interface.

[0098] 10 shows a camera interface 1000 projected by projector subsystem 832 onto the palm of a user's hand 1002. A user can interact with camera interface 1000 to browse digital content, select digital content, or control a digital image capture device of wearable multimedia device 101.

[0099] In some implementations, the camera interface 1000 is configured to display output generated by the wearable multimedia device 101. For example, as described in Figures 11 and 12, the wearable multimedia device 101 can determine exposure timing and exposure parameters for a digital image capture device of the wearable multimedia device 101 to capture one or more images. The wearable multimedia device 101 can provide one or more images to the projector subsystem 832. The projector subsystem 832 can project the camera interface 1000, including a representation of the one or more images, onto a surface, such as the palm of a user's hand 1002, the ground, a wall, a ceiling, or other surface.

[0100] In some implementations, the camera interface 1000 is configured to receive input from a user. For example, a user can provide an input indicating a mode of image capture. The image capture modes can include user-specific modes based on user preferences learned over time, such as from tracked and stored preferences, or general modes designed to capture general subject features of interest based on a database of images and exposure parameters used.

[0101] 11 illustrates an example of a system 1100 for determining image exposure for a camera component of a wearable multimedia device. The system 1100 includes a sensor 1102, a local / network storage 1104, an exposure engine 1122, an image capture device 1124, and an image optimizer 1134. In at least one implementation, one or more operations of the system 1100 are performed by the wearable multimedia device 101. For example, in some implementations, one or more of the sensor 1102, the local / network storage 1104, the exposure engine 1122, the image capture device 1124, or the image optimizer 1134 are operated by or included on the wearable multimedia device 101.

[0102] Sensor 1102 represents a data stream from a sensor on a device implementing system 1100. The sensor 1102 data stream includes data from an interface for a global positioning system (GPS) 1106, an image stream 1108, and data from a user interface 1110. In some implementations, image stream 1108 includes images captured by image capture device 1124. In some implementations, image stream 1108 is a buffer temporarily stored in storage of wearable device 101.

[0103] 8, such as sensors coupled with peripherals interface 806. For example, the sensor 1102 data stream can include data such as motion detection, orientation detection, biometric detection, and depth detection from motion sensor 810, biometric sensor 812, and depth sensor 814. Sensor 1102 data can include data from motion sensor 810 (e.g., accelerometer, rate gyroscope), such as the movement and orientation of wearable multimedia device 101 or other device implementing system 1100.

[0104] In some implementations, the sensor 1102 includes data from the recorder 201. For example, the sensor 1102 can include a video buffer 202, an audio buffer 203, and a photo buffer 204. This data can be used by the system 1100 to determine the timing of one or more exposures as well as exposure parameters. In some implementations, the sensor 1102 represents data obtained from one or more sensors.

[0105] The local / network storage 1104 includes data streams from a calendar 1112, a photo library 1114, and registered images 1116. The calendar 1112 represents data obtained from one or more calendars accessible by the system 1100. For example, the calendar 1112 may represent data for events, including time, location, or associated information, in one or more user calendars.

[0106] The photo library 1114 represents stored images. The photo library 1114 can include images from one or more user databases. The photo library 1114 can include images captured by the image capture device 1124 as well as images captured by other devices. The photo library 1114 can include a database in the storage of the wearable multimedia device 101 or in a storage device communicatively connected to the wearable multimedia device 101, such as a network storage device.

[0107] In some implementations, the photo library 1114 includes information about features in an image. For example, the photo library 1114 may include an image of a person standing in front of a house. The photo library 1114 may include information indicating the identity of the person, the location of the house, the date and time the image was taken, any modifications made to the image, user actions on the image, such as likes and shares, and the exposure parameters used to capture the image, among other things.

[0108] The enrollment image 1116 includes data for generating an exposure. In some implementations, the enrollment image 1116 includes data indicative of a particular object or person desired to be captured in the image. For example, as shown in FIG. 11 , the portion of the enrollment image 1116 in the local / network storage 1104 shows a person 1120 in an image 1118. The image 1118 can be selected by a user of the system 1100. In some implementations, the user selects through the camera interface 1000 which features in an image or which images in a group of images should be used for the enrollment image 1116.

[0109] While the sensors 1102 and local / network storage 1104 are shown with specific elements providing data to the exposure engine 1122, in some implementations one or more of those elements are not used, replaced by other elements, or combined with additional elements. For example, instead of using GPS 1106 data, the system 1100 can use only the image stream 1108 from a camera device, such as image capture device 1124, and the calendar 1112 data. In general, data obtained from sensors and data stores on board the wearable device 101, as well as data obtained from communicatively connected devices, can be used to effect exposure timing and exposure parameter determinations.

[0110] The exposure engine 1122 acquires data from the sensor 1102 and the local / network storage 1104. The exposure engine 1122 determines exposure timing and exposure parameters for an exposure of the image capture device 1124 based on the acquired data. An exposure of the image capture device 1124 can be a first exposure, as well as an exposure subsequent to one or more preceding exposures. For example, the exposure engine 1122 can determine initial exposure settings and exposure timing for generating images in the image stream 1108. The exposure engine 1122 can then determine subsequent exposure parameters and exposure timing based on parsing the image stream 1108 in addition to other data streams in the sensor 1102 and the local / network storage 1104.

[0111] In some implementations, the exposure engine 1122 determines the exposure timing based on image elements. For example, the exposure engine 1122 can parse the image stream 1108 to determine one or more elements of one or more images, such as lighting, distance to objects, object type, person identity, among others. The exposure engine 1122 can determine whether to save an image in the image stream 1108 based on one or more elements of the one or more images.

[0112] In some implementations, the image capture device 1124 captures images provided in the image stream 1108. For example, the image capture device 1124 may receive a signal from the wearable device 101 or other device implementing the system 1100 to begin capturing exposures. The captured exposures may be temporarily stored in a buffer of the image stream 1108. This buffer may be provided to the exposure engine 1122 to determine which of the exposures in the stream 1108 should be saved to storage, as well as adjustments to exposure parameters or exposure timing.

[0113] In some implementations, the exposure engine 1122 determines when to start collecting exposures of the image capture device 1124. For example, the exposure engine 1122 can obtain accelerometer data, location data, or other data from the sensor 1102 of the device implementing the system 1100, such as the wearable device 101. The exposure engine 1122 can obtain data from the local / network storage 1104. The exposure engine 1122 can determine when to start collecting exposures based on one or more of the data from the sensor 1102, the data from the local / network storage 1104, a particular user's preferences, general user preferences generated from several users of similar devices or several similar users, among others.

[0114] In some implementations, the exposure engine 1122 determines to initiate image capture of the image capture device 1124 based on data from the local / network storage 1104. For example, the exposure engine 1122 may identify a calendar event showing a track meet. The calendar event may include a location. The calendar event may include identities of people racing in the track meet. The exposure engine 1122 can identify the identities of the people based on the information in the calendar event. The exposure engine 1122 can parse the photo library 1114 to find photos of the people or parse a database of features to identify which features of the image correspond to the identities of the people.

[0115] In some implementations, the exposure engine 1122 identifies the image 1118 after determining that the person 1120 is associated with a calendar event. For example, the exposure engine 1122 may determine that the calendar event mentions the person 1120 by name, or that the location is a known location that corresponds to the person 1120, such as a school, or that the person 1120 is a known person that is always captured when within the field of view of the image capture device 1124.

[0116] 11, the exposure engine 1122 determines to trigger an exposure based on comparing a location of the GPS data stream of the sensor 1102 to a location of a calendar event in the calendar data stream 1112. In some implementations, the exposure engine 1122 determines to trigger an exposure based on an accelerometer data stream indicating that a user has placed a device corresponding to the image capture device 1124, such as the wearable device 101, in a location relevant for image capture. The exposure engine 1122 can compare a current data stream, such as the accelerometer data stream, to one or more historical data streams or provide the one or more data streams to a trained machine learning model to determine when to trigger an exposure.

[0117] In some implementations, the exposure engine 1122 determines the exposure parameters before triggering the exposure. For example, the exposure engine 1122 can determine an initial International Organization for Standardization (ISO) number, shutter speed, or aperture, or any suitable combination thereof, of the image capture device 1124. In some implementations, the exposure engine 1122 obtains a data stream to determine which starting parameters to use. For example, the exposure engine 1122 can obtain one or more of a time of day, a light intensity value indicating the light intensity of a light detector of a device implementing the system 1100, such as the wearable device 101, among others.

[0118] In some implementations, the exposure engine 1122 compares one or more data values ​​indicative of the acquired data to historical data. For example, the exposure engine 1122 can identify a time range that corresponds to the current time based on the current time, one or more time ranges, and a comparison of the current time to the one or more time ranges. The time ranges can also correspond to typical exposure parameters, such as lower ISO during the day and higher ISO at night.

[0119] In some implementations, the exposure engine 1122 provides one or more data values ​​indicative of the acquired data to the trained model. For example, the exposure engine 1122 can include the trained model or can be communicatively connected to a device that executes the trained model. In some implementations, the trained model is a cloud server. For example, the exposure engine 1122 can be communicatively connected to a cloud computing server that executes the trained model.

[0120] The trained model can be a type of machine learning model trained to use the data stream data to generate exposure settings that result in high quality images. High quality images can include images that match images marked as good looking by humans or automated systems. Features can be extracted from these known high quality images to train the model of the exposure engine 1122 to generate parameters and exposures that are likely to result in high quality images, such as images that are not blurry, that are not too dark or bright, that focus on a person or object or a specific object or person, that are framed such that objects are spatially positioned in a way that matches images marked as high quality, among others.

[0121] In some implementations, the image optimizer 1134 optimizes one or more captured images. For example, the image optimizer 1134 can capture one or more images from the image capture device 1124 based on the exposure parameters and exposure timing of the exposure engine 1122. The image optimizer 1134 can include one or more trained models for generating one or more high-quality images based on the captured one or more images. In some implementations, the image optimizer 1134 selects one or more images from a set of one or more captured images. For example, the image optimizer 1134 can provide one or more captured images to one or more trained models. The one or more trained models can be trained to determine high-quality images based on ground truth data indicating images marked as high quality by a human or an automated process. The one or more trained models can generate an output indicating the top one or more images that most closely represent features associated with known high-quality images, such as unblurred objects, proper lighting, among others.

[0122] In some implementations, the image optimizer 1134 generates at least one image based on a set of one or more captured images. For example, the image optimizer 1134 can combine one or more captured images to generate a high-quality image. In some cases, the image optimizer 1134 is configured to provide one or more captured images to a trained model. The model can be trained to generate one or more images of high quality based on the set of one or more captured images. As discussed herein, the model can be trained with ground truth data indicating images marked as high quality by a user or an automated process. The model can combine elements of multiple images to create a composite image.

[0123] In some implementations, the model determines regions of one or more captured images that correspond to a high quality image and combines the regions into a composite image for output. The model can also process one or more captured images or the generated composite image and perform adjustments to generate a final image that exhibits the characteristics of the high quality image. The adjustments can include interpolation to increase resolution, cropping, color balancing, lighting effects, among others. In some cases, the model can determine that the high quality image characteristics of the final image satisfy a threshold that indicates an image of sufficient quality to be captured, stored on the wearable device 101, or viewed on the wearable device 101.

[0124] In some implementations, the exposure engine 1122 obtains an output from the trained model. As discussed above, the exposure engine 1122 can provide data to the trained model. The trained model can provide an output to the exposure engine 1122 indicating one or more parameters or timing for image capture. For example, based on providing data including time of day, location, and person of interest, the trained model can provide an output indicating exposure parameters including ISO, shutter speed, and aperture, as well as the timing of exposure. The timing can include the frequency of capturing images. In some cases, the trained model can detect a feature, such as a person 1120, in image data, such as the image stream 1108 provided as input by the image capture device 1124, and determine the timing of exposure as when the person 1120 is represented in the image data.

[0125] In some implementations, the exposure engine 1122 obtains data from other data streams. For example, the exposure engine 1122 can obtain data from a weather data stream indicating the weather for one or more locations. The exposure engine 1122 can determine current weather conditions based on the current location. The exposure engine 1122 can provide the current weather conditions to a trained model discussed herein or compare the data with historical data to determine exposure parameters. For example, if the weather forecast or current conditions indicate a cloudy day, this can result in exposure parameters including a higher ISO being determined.

[0126] In some implementations, the exposure engine 1122 uses multiple modes based on scenario variables. For example, the exposure engine 1122 can determine that the network connection is off. The exposure engine 1122 can process the data stream offline and determine exposure timing and exposure parameters without using network resources. In some implementations, the offline processing includes comparing the acquired data to historical data. In some implementations, the offline processing includes providing the data to an offline implementation of a machine learning model. In some implementations, the offline processing includes not providing the data, at least in part, to an online implementation of a machine learning model running on a device communicatively connected to a device implementing the system 1100, such as the wearable device 101.

[0127] In some implementations, the exposure engine 1122 detects a partial gesture of the user. For example, the user can raise a hand or perform another gesture associated with taking a picture with the image capture device 1124. The gesture can be configured to launch a camera application or trigger a camera exposure. The exposure engine 1122 can detect the beginning of the gesture and trigger exposure before the gesture is completed, such that when the gesture is completed and the camera function of the associated camera application is performed, or thereafter, the exposure engine 1122 has determined appropriate exposure parameters for the scene. In this manner, the system 1100 and the exposure engine 1122 can increase the efficiency of image capture by preemptively optimizing parameters based on one or more data streams of the sensor 1102 and the storage 1104.

[0128] In some implementations, the wearable device 101 is worn on the user facing the direction the user is facing. For example, as shown in FIG. 11, the image capture device 1124 captures an image 1126 at a location and time, in this case at a track meet. Prior to capturing the image 1126, the exposure engine 1122 provides the image capture device 1124 with data indicative of exposure parameters. The exposure engine 1122 determines that the user is at the track meet based on the calendar data stream 1112 and the GPS data stream 1106. The exposure engine 1122 also determines that the person 1120 is a person of interest.

[0129] In some implementations, an image capture device 1124 provides images to the image stream 1108. For example, an exposure engine 1122 can acquire images as they are captured by the image capture device 1124 and stored in a buffer of the image stream 1108. The exposure engine 1122 can determine adjustments to exposure parameters of the image capture device 1124 based on the image stream 1108. The exposure engine 1122 can determine which parameters, or ranges of values ​​of parameters, are likely to result in high quality images as discussed herein based on the data streams of the sensor 1102 and the storage 1104.

[0130] In some implementations, the exposure engine 1122 tunes exposure parameters based on current data provided by the camera device. For example, the exposure engine 1122 can tune exposure parameters of the image capture device 1124 based on the image stream 1108. In some implementations, the exposure engine 1122 provides data based on the data of the image stream 1108 to the trained model. In some implementations, the model is trained to determine whether an image is of high quality. For example, the model can extract features of images in the image stream 1108 and determine based on the extracted features compared to historical training data that indicates features associated with high quality images, such as in-focus images, aligned images, properly lit images, and low quality images, such as blurry images, too dark images, too bright images, misaligned images, or images with poor object placement in the scene, among others.

[0131] In some implementations, the image 1126 is generated by the exposure engine 1122 based on data provided by the image capture device 1124. For example, the exposure engine 1122 can select the image 1126 from a buffer of images acquired by the image capture device 1124 based on one or more processes. In some implementations, the one or more processes include processing a trained model. For example, the exposure engine 1122 can provide one or more images of the image stream 1108, along with one or more items of data from the data streams 1102 and 1104, to the trained model. The trained model can provide an output indicating whether a given image is of high quality or whether the image should be saved or provided to a user for viewing.

[0132] In some implementations, the image optimizer 1134 acquires the image 1126. For example, the image optimizer 1134 can acquire the image 1126 from the image capture device 1124 as well as the exposure engine 1122. In some implementations, the operation of the image optimizer 1134 is performed on a device communicatively connected to the device implementing the system 1100. In some implementations, the operation of the image optimizer 1134 is performed on the device implementing the system 1100. In some implementations, the image optimizer 1134 performs one or more adjustments on the acquired image 1126. For example, the image optimizer 1134 can detect the person 1120 and crop the image 1126 to focus on the person 1120. The image optimizer 1134 can crop out other figures, such as person 1128, and defocus other figures, such as person 1130, to generate an image that includes only the person 1120. An image optimizer 1134 can combine multiple exposures to improve the dynamic range or other characteristics of the final composite image.

[0133] In some implementations, the exposure engine 1122 learns exposure timing based on the context of the scene. For example, in the track meet shown in image 1126, a person 1120 is running around a track. The exposure engine 1122 can determine how often the person 1120 approaches the same area of ​​the track based on the person 1120 being repeatedly observed as the person 1120 runs around the track. The exposure engine 1122 can modify the exposure to increase the frequency of captures when the person 1120 is expected to be in an area of ​​the track within the field of view of the wearable device 101 based on pace and frequency.

[0134] In some implementations, the exposure engine 1122 obtains historical data to determine the frequency of capture. For example, the exposure engine 1122 can obtain data of the pace of the person 1120 and determine that the pace corresponds to a frequency indicating when the person 1120 is in the field of view of the wearable device 101 during a portion of the track that is in the field of view of the wearable device 101. The exposure engine 1122 can send a signal to the image capture device 1124 configured to increase image capture when the person 1120 is expected to be in the field of view of the image capture device 1124. The exposure engine 1122 can send a signal to the image capture device 1124 configured to decrease image capture when the person 1120 is not expected to be in the field of view of the image capture device 1124.

[0135] In some implementations, the exposure engine 1122 takes images from the image stream 1108 and identifies a level of movement in the scene. For example, the exposure engine 1122 can compare a first image in the image stream 1108 to a subsequent image in the image stream 1108 and identify an amount of difference between the images. If the amount of difference, indicating a difference in the object's position or pixel values, satisfies a threshold, the exposure engine 1122 can send a signal to the image capture device 1124 configured to increase image capture to capture movement. In a track scene, this can result in an image being captured when a runner is in the field of view, thus causing a significant difference in the subsequent images stored in the image stream 1108.

[0136] For other scenes, such as views of animals, birthday parties, dashboard camera scenes, among others, the exposure engine 1122 can be configured to increase image capture of the image capture device 1124 to coincide with greater levels of movement. Because greater levels of movement typically coincide with interesting or noteworthy events, the system 1100 efficiently captures interesting or noteworthy events in the field of view of the wearable device 101 by adjusting the timing of exposure to increase during periods of increased movement, while minimizing storage and processing power by reducing image capture during periods of decreased movement. In some implementations, increased and decreased movement is determined relative to the average level of movement in the scene over a period of time.

[0137] After capturing an image, the image capture device 1124 or the exposure engine 1122 can provide the image to an image optimizer 1134. The image optimizer generates an optimized version of one or more captured images. The optimized version can include post-processing adjustments such as adjustments to lighting, cropping, color, and focus, among others. In some implementations, the image optimizer 1134 provides the images for display and selects the images for storage. For example, the image optimizer 1134 can determine that one or more images are of high quality or that one or more composite images are of high quality. The high quality images can be stored or provided for viewing as discussed herein. Images that are not selected or that were used to generate a composite image can be removed from storage. In this manner, the system 1100 allows for the capture of many images while minimizing the storage and processing impact associated with storing all of the images in non-transitory storage.

[0138] In some implementations, the system 1100 provides images captured by the image capture device for storage. For example, the image capture device 1124 can generate multiple images of a track meet, or any other event or scene. The system 1100 can include an interface with a server storage device or an on-board storage device. Images captured over a past period of time, such as over a day, over an hour, or over the length of an event, such as an event shown in a calendar, can be stored at a higher resolution. For example, all images captured over that period of time can be stored at least temporarily to provide a user with an opportunity to review the stored images, such as images optimized by the image optimizer 1134, and to ensure that the system 1100 did not miss any important events. After that period of time, the system 1100 or a connected storage device can remove the image data from storage.

[0139] The final image generated by the image optimizer 1134 is provided for display or storage. In the example of FIG. 11, the image optimizer 1134 provides an optimized version of the image 1126 to the projector subsystem 832 of the wearable device. As discussed in FIG. 10, the projector subsystem 832 can project the optimized version of the image 1126 onto a surface for viewing by a user of the system 1100. In some implementations, the image optimizer 1134 provides the optimized version of the image 1126 to another device for viewing. For example, the image optimizer 1134 can provide the optimized version of the image 1126 to a smartphone, laptop, or desktop computer for viewing on an LCD, LED, OLED, or other form of conventional screen.

[0140] 12 illustrates an example of a system 1200 for determining exposure parameters and generating an image exposure. As illustrated, the system 1200 includes a wearable device 101 and a network server 1202. The wearable device 101 includes a device database 1204, a device sensor 1206, and an exposure engine 1122. The exposure engine 1122 controls, in part, one or more of the exposure parameters ISO 1208, aperture 1210, and shutter speed 1212 to generate an exposure 1214. An example exposure generated based on the exposure parameters of the exposure engine 1122 is shown in FIG. 11 as image 1126.

[0141] The network server 1202 provides a database including data such as, among others, calendar event data, data from cloud computing, cloud computing machine learning models, user profile data, data from the Internet, data from a user's online database such as an online image library, etc. The wearable device 101 obtains the data provided by the network server 1202. In particular, the exposure engine 1122 obtains the data from the network server 1202 to determine exposure parameters and timing of exposure, as discussed with reference to FIG.

[0142] The device database 1204 and the device sensors 1206 provide the exposure engine 1122 with additional data for determining one or more exposure parameters or exposure timing. In some implementations, the exposure engine 1122 obtains data from the device database 1204 and the device sensors 1206 and provides the data to the network server 1202 for processing. The exposure engine 1122 can then obtain an output from the network server 1202 indicating the results of the processing. For example, the exposure engine 1122 can provide the data from the device database 1204 and the device sensors 1206, as well as data stored in a database connected to the network server 1202, to the network server 1202. The network server 1202 can provide the data to one or more cloud computing architectures including one or more trained models. The exposure engine 1122 can then obtain an output from the cloud computing system to determine one or more exposure parameters or exposure timing.

[0143] In the example of FIG. 12, the exposure engine 1122 obtains data from the device database 1204, the device sensor 1206, and the network server 1202. In some implementations, the exposure engine 1122 processes the data on-board using on-board algorithms or trained models. In some implementations, the exposure engine 1122 processes the data using cloud computing provided by a connection to the network server 1202. The exposure engine 1122 determines one or more exposure parameters based on the processing. The exposure parameters include ISO 1208, aperture 1210, and shutter speed 1212. The exposure engine 1122 applies the exposure parameters to a camera device, such as an image capture device 1124.

[0144] In some implementations, the exposure engine 1122 applies the exposure parameters to the camera device using an interface of the wearable device 101. For example, the wearable device 101 can be configured to control the image capture device 1124. The exposure engine 1122 can provide the exposure parameters and exposure timing to the wearable device 101's interface with the image capture device 1124. The wearable device 101 can then apply the exposure parameters and exposure timing generated by the exposure engine 1122 to the image capture device 1124.

[0145] After the exposure engine 1122 determines the exposure parameters and exposure timing for one or more exposures, it may generate an exposure 1214. The exposure 1214 may result in an image, such as an image captured by the image capture device 1124.

[0146] In some implementations, the exposure engine 1122 is operated by the network server 1202. For example, the wearable device 101 can establish a connection with the network server 1202. The wearable device 101 can provide the network server 1202 with data corresponding to the device database 1204, such as captured image data, stored information, and data from the device sensor 1206, among other things. The exposure engine 1122 can operate on the network server 1202 and generate exposure parameters and exposure timing. As discussed herein, the exposure engine 1122 can process data, including data from a network database connected to the network server 1202, using algorithms and trained models. The exposure engine 1122 can then generate and provide the exposure parameters and exposure timing to the wearable device 101. In some implementations, the exposure engine 1122 provides the exposure parameters and exposure timing to the network server 1202. For example, network server 1202 can obtain exposure parameters and exposure timing and provide the exposure parameters and exposure timing to wearable device 101. Wearable device 101 can then control a camera device, such as image capture device 1124, to apply the exposure parameters and generate exposure 1214, as discussed above.

[0147] In some implementations, the exposure engine 1122 determines exposure parameters based on a classification of the scene. For example, when the exposure engine 1122 detects a fast moving object, such as a running dog, the exposure engine 1122 can increase the shutter speed. Other context models, such as birthday candles, can be applied to the devices based on a determination that the scene includes features that match one or more existing context models. The context models can be trained to provide appropriate exposure parameters and exposure timing for a particular scene. In general, multiple context models can be stored on the device or in a network server, such as the network server 1202, and applied when the exposure engine 1122 is processing data. During processing, the exposure engine 1122 can apply the context models to determine exposure parameters for a particular scene.

[0148] As an illustrative example, the wearable device 101 may be used during a whale watching tour. In this case, the exposure engine 1122 may include a context model for whale watching. This context model may detect changes in an image stream, such as image stream 1108, that indicate a whale is breaching the surface. This context model may be tuned to be sensitive to certain features, such as the appearance of white water or foam in a body of water, which generally indicates a breach. In this manner, the context model may be fine-tuned by a user of the system or programmatically by determining the similarity of multiple captured images and the associated exposure parameters that correspond to high quality images. That is, data from multiple devices may be combined to determine which exposure parameters and exposure timings work best in a particular scenario. In some implementations, this data is captured in training a model, where the model recognizes the context based on the captured data and generates a set of exposure parameters that match previously captured high quality images in a similar context.

[0149] In some implementations, the exposure engine 1122 retrieves previous exposure parameter data. For example, a user may select an object or subject displayed on the camera interface 1000. The exposure engine 1122 may generate exposure parameters to focus on the object or subject and adjust the ISO, aperture, and shutter speed based on the lighting conditions and the subject or object's movement to clearly capture the object or subject. A system such as the system 1200 may record the exposure parameters corresponding to the object or subject for later use. For example, the exposure engine 1122 may identify a previously selected person based on data from the image stream 1108. The person may have associated exposure parameters. The exposure engine 1122 may query whether there are exposure parameters corresponding to the person based on identifying the person in an image in the image stream 1108.

[0150] In some implementations, the exposure engine retrieves a recorded exposure pattern for a subject or object. For example, the exposure engine 1122 can retrieve previously recorded exposure parameters for a particular person. The exposure engine 1122 can apply those exposure parameters to the image capture device 1124 for subsequent image capture. In this manner, a person or object can be focused on when the person or object is in the field of view of the image capture device 1124 or when the object is in the field of view of the image capture device 1124. The person or object may exit the field of view. When the person or object re-enters the field of view after a period of time, the exposure engine 1122 can identify and apply one or more values ​​corresponding to the previously applied exposure parameters.

[0151] Further Exemplary Processes 13 is a flow diagram of an example process 1300 for generating an image exposure based on determined exposure parameters. In some implementations, the process 1300 is performed using the wearable multimedia device 101 described with reference to FIGS.

[0152] The process 1300 includes accessing (1302) information from one or more data streams, where the one or more data streams include at least one of image data or pre-set image parameters. For example, as shown in FIG. 11, an image capture device 1124, such as a camera, of the wearable device 101 can obtain data from the sensor 1102 and local / network storage 1104. The data from the sensor 1102 and local / network storage 1104 can include data from the GPS 1106, image stream 1108, a user interface 1110, such as a camera interface 1000, calendar data 1112, photo library 1114, and registered image 1116.

[0153] The process 1300 includes determining (1304) exposure parameters for capturing one or more images using information from the one or more data streams. For example, an exposure engine 1122 can determine an exposure pattern for an image capture device 1124, such as a camera of the wearable device 101, based on data from the sensor 1102 and local / network storage 1104.

[0154] The process 1300 includes applying 1306 the determined exposure parameters to a camera associated with the wearable device. For example, the exposure engine 1122 can provide the exposure parameters determined based on data from the sensor 1102 and local / network storage 1104 to an image capture device 1124, or a control device such as the wearable device 101.

[0155] The process 1300 includes controlling 1308 the camera to capture one or more images with the exposure set with the applied exposure parameters. For example, the image capture device 1124 can capture the image 1126 based on the exposure parameters provided by the exposure engine 1122. The exposure engine 1122 can send a control signal directly to the image capture device 1124 to capture the image 1126. The exposure engine 1122 can send the control signal to the wearable device 101 configured to control the image capture device 1124 to capture the image 1126.

[0156] The features described herein can be implemented in digital electronic circuitry, or in computer hardware, computer firmware, computer software, or in combinations of them. The features can be implemented as a computer program product tangibly embodied in an information carrier, for example a machine-readable storage device, for execution by a programmable processor. Method steps can be performed by the programmable processor executing a program of instructions for performing functions of the implementations described herein by operating on input data and generating output.

[0157] The features described herein may advantageously be implemented as one or more computer programs executable on a programmable system including at least one programmable processor coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to send data and instructions to the data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform a particular activity or bring about a particular result. A computer program may be written in any form of programming language, including compiled or interpreted languages ​​(e.g., Objective C, Java), and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0158] Processors suitable for executing a program of instructions include, by way of example, both general purpose and special purpose microprocessors, and the sole processor or one of multiple processors or cores of any kind of computer. Generally, a processor receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer can communicate with mass storage devices for storing data files. These mass storage devices can include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices, such as EPROMs, EEPROMs, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by or incorporated in ASICs (Application Specific Integrated Circuits). To enable user interaction, features may be implemented on a computer having a display device, such as a CRT (cathode ray tube), LED (light emitting diode), or LCD (liquid crystal display) display or monitor, for displaying information to the creator, and a keyboard and pointing device, such as a mouse or trackball, by which the creator can input into the computer.

[0159] One or more features or steps of the disclosed embodiments may be implemented using an application programming interface (API). An API may define one or more parameters passed between a calling application and other software code (e.g., an operating system, a library routine, a function) that provides a service, provides data, or performs an operation or calculation. An API may be implemented as one or more calls in program code that send or receive one or more parameters through a parameter list or other structure based on a calling convention defined in an API specification. A parameter may be a constant, a key, a data structure, an object, an object class, a variable, a data type, a pointer, an array, a list, or another call. API calls and parameters may be implemented in any programming language. A programming language may define the vocabulary and calling conventions used by a programmer to access functions that support the API. In some implementations, an API call may report to an application the capabilities, such as input capabilities, output capabilities, processing capabilities, power capabilities, communication capabilities, etc., of the device on which the application runs.

[0160] Several implementations have been described above. However, it will be understood that various modifications may be made. Elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. In yet another example, the logic flow depicted in the figures does not require the particular order or sequence depicted to achieve desirable results. In addition, other steps may be provided or steps may be eliminated from the described flow, and other components may be added to or removed from the described systems. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0161] 101 Wearable multimedia devices, wearable devices 102 Cloud Computing Platform 103 Network 104 Third Party Application ("App") Developers 105 Third Party Platforms 106 Database 200 Data Processing System 201 Recorder 202 Video Buffer 203 Audio Buffer 204 Photo Buffer 205 Import Server 206 Datastore 207 Video Processor 208 Audio Processor 209 Photo Processor 210 Third Party Processors 211 Image processing / editing applications 212 Image processing / editing applications 213 Speech to Text Conversion Application 214 Speech to Text Applications 215 Image Processing Applications 216 Image Processing Applications 217 Applications 300 Data Processing Pipeline 301 Import Server 302 Video Processor 304 Recorder 305 Audio Processor 306 Apps 307 Video Processing App 308 Data Merge Process 400 Data Processing Pipeline 401 Import Server 402 Third Party Processors 404 Recorder 405 Audio Processor 406 App 407 Third Party Applications 408 User Callback Queue 600 processes 700 Architecture 702 Processors, Devices 704 Storage Devices 706 Network Interfaces, Devices 708 Computer-readable media, devices 710 Communication Channel 712 Operating Systems 714 Network Communication Module 716 Data Processing Instructions 718 Interface Instructions 800 Architectures, Devices, and Systems 802 Memory Interface 804 Central Processing Unit, Processor 806 Peripheral Device Interface 810 Motion Sensor 812 Biometric Sensors 814 Depth Sensor 815 Location Processor 816 Electronic Magnetometer 817 Environmental Sensor 820 Camera Subsystem, Camera / Video Subsystem 822 Optical Sensor 824 Communication Subsystem 826 Audio Subsystem 828 Speaker 830 Microphone 832 Projector Subsystem, Projection Subsystem 840 I / O Subsystem 842 Touch Controller 844 Another input controller, other input controller 846 Touch Surface 848 Other Input / Control Devices 850 Memory 852 Operating Systems 854 Communication Order 858 Sensor Processing Instructions 860 Recorder Command 901 Controller 902 Battery 903 Power Management Chip (PMIC) 904 Solid-state laser 905 XY Scanner 906 Driver 907 Memory 908 Digital to Analog Converter (DAC) 909 Analog-to-Digital Converter (ADC) 1000 Camera Interface 1002 User's palm 1100 System 1102 Sensor, data stream 1104 Local / Network Storage, Data Stream 1106 Global Positioning System (GPS), GPS data stream 1108 Image Stream 1110 User Interface 1112 Calendar, Calendar Data Stream, Calendar Data 1114 Photo Library 1116 Registered Images 1118 images 1120 People 1122 Exposure Engine 1124 Image capture device 1126 images 1128 People 1134 Image Optimizer 1200 System 1202 Network Server 1204 Device Database 1206 Device Sensor 1208 ISO 1210 Aperture 1212 Shutter Speed 1214 Exposure 1300 Processes

Claims

1. accessing information from one or more data streams at the wearable device, the one or more data streams including at least one of image data or pre-set image parameters available to the wearable device; using the information from the one or more data streams to determine exposure parameters for capturing one or more images; applying the determined exposure parameters to a camera associated with the wearable device; controlling the camera to capture one or more images using an exposure set according to the applied exposure parameters; A method comprising:

2. receiving an input to take a picture using the camera; or Predicting that the camera will be activated to take a photo based on a context of the wearable device determined using one or more sensors.

2. The method of claim 1, comprising the step of: accessing the information from the one or more data streams in response to

3. determining the exposure parameters further comprises: adjusting the pre-set image parameters based on target features determined from the one or more data streams.

2. The method of claim 1, comprising:

4. The method of claim 3 , wherein the pre-set image parameters are generated based on at least one of context, calendar information, or location information of the wearable device determined using one or more sensors.

5. identifying features within the one or more data streams; detecting the feature in the image data of the one or more data streams; 2. The method of claim 1, comprising:

6. The method of claim 5 , wherein the features include at least one of a representation of a particular person or a representation of a particular object.

7. Controlling the camera to adjust one or more lenses to focus on the feature.

6. The method of claim 5, comprising:

8. Controlling the camera to capture the feature in the one or more images.

6. The method of claim 5, comprising:

9. controlling the camera to capture the feature in one or more additional images using the exposure set with an adjusted version of the applied exposure parameters.

9. The method of claim 8, comprising:

10. The method of claim 1 , wherein the one or more data streams include at least one of calendar information, location information, or stored image data.

11. One or more computers; one or more computer memory devices interoperably coupled to the one or more computers and having a tangible, non-transitory, machine-readable medium storing one or more instructions, which, when executed by the one or more computers, accessing, at a wearable device, information from one or more data streams, the one or more data streams including at least one of image data or pre-set image parameters available to the wearable device; using the information from the one or more data streams to determine exposure parameters for capturing one or more images; applying the determined exposure parameters to a camera associated with the wearable device; and Controlling the camera to capture one or more images using an exposure set according to the applied exposure parameters. one or more computer memory devices for performing one or more operations including A computer-implemented system comprising:

12. The operation, receiving an input to take a picture using the camera; or Predicting that the camera will be activated to take a photo based on a context of the wearable device determined using one or more sensors.

12. The system of claim 11, further comprising: responsively accessing the information from the one or more data streams.

13. determining the exposure parameters adjusting the pre-set image parameters based on target features determined from the one or more data streams. The system of claim 11 , comprising:

14. 14. The system of claim 13, wherein the pre-set image parameters are generated based on at least one of context, calendar information, or location information of the wearable device determined using one or more sensors.

15. The operation, identifying features within the one or more data streams; detecting the feature in the image data of the one or more data streams; The system of claim 11 , comprising:

16. The system of claim 15 , wherein the features include at least one of a representation of a particular person or a representation of a particular object.

17. The operation, Controlling the camera to adjust one or more lenses to focus on the feature. The system of claim 15, comprising:

18. The operation, Controlling the camera to capture the feature in the one or more images. The system of claim 15, comprising:

19. The operation, controlling the camera to capture the feature in one or more additional images using the exposure set with an adjusted version of the applied exposure parameters; 20. The system of claim 18, comprising:

20. One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to: accessing, at a wearable device, information from one or more data streams, the one or more data streams including at least one of image data or pre-set image parameters available to the wearable device; determining exposure parameters for capturing one or more images using the information from the one or more data streams; applying the determined exposure parameters to a camera associated with the wearable device; and controlling the camera to capture one or more images using an exposure set according to the applied exposure parameters; One or more non-transitory computer-readable media for performing operations including:

Images