Displaying images using a wearable multimedia device

Abstract

A system, method, device, and non-transitory computer-readable storage medium are disclosed for a wearable multimedia device and a cloud computing platform with an application ecosystem for processing data captured by the wearable multimedia device. In one embodiment, operations performed by the wearable multimedia device or the cloud computing platform include obtaining an image captured using a camera coupled to the wearable multimedia device, comparing the image to one or more known images, identifying one or more target features in the image, generating a new image that includes a portion of the image that includes the one or more target features, and providing the new image to the wearable multimedia device for presentation.

Description

[Technical field]

[0001] The present disclosure relates generally to wearable multimedia devices, and in particular to displaying images using wearable multimedia devices. [Background technology]

[0002] High-precision laser scanners (e.g., MEMS scanners) have been developed that can turn any surface into a virtual interface (VI). For example, a laser-projected VI can be projected onto the palm of a user's hand or other surfaces. Such VIs can be used with electronic devices. Summary of the Invention [Means for solving the problem]

[0003] Disclosed are systems, methods, devices, and non-transitory computer-readable storage media for displaying and manipulating images using a wearable multimedia device and associated network architecture. In some implementations, a user uses a laser projected VI associated with the wearable multimedia device to view images, interact with the displayed images, or control an image capture device, or any suitable combination thereof. The image capture device can be included within the wearable multimedia device.

[0004] In some implementations, interacting with the image includes cropping the image. The cropping can be directed based on a reference image selected by a user or an automated process. The cropping can include identifying one or more portions of the image that include features similar to features selected by the user or the automated process in the reference image. The cropped portions of the image can be displayed using a VI of the wearable multimedia device.

[0005] In general, a wearable multimedia device may include a projector subsystem configured to visually present information to a user in the form of projected light. For example, the projector subsystem may project light onto a surface (e.g., a surface of a user's hand, such as the palm of the user's hand) according to a particular spatial and / or temporal pattern such that the user perceives a VI with one or more user interface elements. Furthermore, the user may perform gestures to interact with the VI.

[0006] In some implementations, the wearable multimedia device can use VIs to present a camera interface that allows a user to view or interact with captured images, as well as capture images or select, like, or dislike one or more images with predefined gestures. This VI can also be referred to as a camera interface.

[0007] In some implementations, a wearable multimedia device is associated with a networked infrastructure that includes hardware and / or software components for generating one or more cropped images based on one or more images captured by an image capture device (e.g., a camera) included in the wearable multimedia device. In some implementations, those components include one or more network devices that receive data from the wearable device and generate one or more cropped images based on the images captured by the image capture device. For example, one such network device can include a processor for identifying one or more portions of an image to be cropped and comparing the one or more identified portions to a known high-quality image. The processor can perform these operations by implementing instructions to extract one or more features from the identified portion of the image being processed and one or more features from the image used as a reference. The processor can compare the features of the identified portion to the high-quality image. The processor can select the identified portion based on a comparison, such as a comparison that satisfies a threshold, and generate a cropped image based on the content within the selected portion of the image.

[0008] In some implementations, the networked infrastructure includes a cloud computing architecture, and the network device can be a back-end network device connected to the wearable multimedia device via a network connection, e.g., the network device can be a network server, which is connected to the wearable multimedia device using a wired or wireless network.

[0009] In some implementations, the hardware and software components include one or more processors on the wearable multimedia device itself and instructions encoded in a storage medium. For example, a wearable multimedia device may include a processor and a program used to generate a cropped image.

[0010] In some implementations, the components for generating the cropped image include a cropping engine and a target feature engine. The cropping engine, or the target feature engine, or both, may be implemented by programming with software instructions and executing those instructions using one or more processors. In some cases, the cropping engine, or the target feature engine, or both, may be implemented using a dedicated processor, for example, realized using an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), or other suitable hardware.

[0011] In the following sections, for ease of explanation, the innovative techniques are described in the context of a wearable multimedia device that includes an image capture device component (e.g., a camera) and a network server communicatively coupled to the wearable multimedia device, the network server including a processor and a cropping engine and a target feature engine implemented by executing instructions using the processor. However, the techniques are equally applicable to other hardware and / or software components associated with a wearable multimedia device, including, for example, the components described above. For example, the techniques are applicable to implementations in which the processor, cropping engine, and target feature engine are in a wearable multimedia device that also includes an image capture device.

[0012] In some implementations, a target feature engine can obtain one or more target images selected by a user or an automated process and generate one or more features based on the selected target images. The target feature engine can provide the generated features to a cropping engine. The cropping engine can obtain a new image and determine which portions of the new image should be cropped based on the features. In general, the cropping engine can identify portions of an image that contain similar features compared to the selected target images.

[0013] In some implementations, the processor used to generate the cropped image acquires data from one or more of the sensors mounted on the wearable multimedia device or from other connected network devices, such as a network database storage device. The processor can generate one or more cropped images based on the acquired data. Depending on the implementation, the processor can execute instructions corresponding to a machine learning model. The machine learning model (also simply referred to as a model) can include one or more of an offline trained model or a trained model in a cloud computing architecture. Additionally or alternatively, the processor can execute instructions corresponding to an offline algorithm, or instructions corresponding to a cloud-based program algorithm, or a combination of an offline trained model, a trained model in a cloud computing architecture, and an offline or cloud-based program algorithm.

[0014] In some implementations, the processor provides the images to a model trained to determine the quality of the images. For example, the model can be trained using training data that indicates images marked by a user or algorithm as high quality, such as images that are in focus, prominently feature people, focus on objects, and show natural lighting and coloring, among others, and images marked by a user or algorithm as low quality, such as images that are out of focus, poorly focused, colored, or lit, among others. The model can also be trained based on a set of images of a user. The model can be trained to generate images with similar characteristics compared to a set of images selected by a user, a trained model, or an algorithm.

[0015] Implementations described herein may provide various technical advantages. As an example, a system that uses one or more back-end network server processors in a networked infrastructure to generate cropped images may reduce processing or storage requirements, or both, for a wearable multimedia device. The system may also enhance the quality of images by automatically acquiring images that include notable or interesting events according to the preferences of one or more users. For example, the system may enhance the quality of images acquired for a user by cropping portions of a larger image that may include one or more features that do not match the reference image to include objects or subjects that have features that match those of a reference image, such as a reference image selected by one or more users, thereby reducing the visibility of unsightly or otherwise undesirable features in the captured image and focusing on features that appear in the reference or target image.

[0016] In some cases, the disclosed techniques can reduce the storage required for an image by allowing storage of a portion of the image. For example, instead of storing the entire image captured by an image capture device, only a cropped portion containing the content of interest can be stored. In this way, less storage can be used in the wearable multimedia device, or in the backend server, or both. A system implementing the disclosed techniques can reduce the storage requirements of the wearable device while retaining high quality images by reducing the size of the image through cropping.

[0017] In some cases, the disclosed techniques can reduce the processing power consumption of a wearable device: for example, the system can reduce the size of a captured image to only the portions of the image that depict objects or people of interest and notable or interesting events, thereby reducing the processing power required to process the entire image upon subsequent viewing, sharing, or storage.

[0018] In some cases, the disclosed techniques can also save time that would otherwise be spent adjusting captured images. For example, the disclosed techniques can automatically crop images, thereby saving time that would otherwise be spent manually processing images to retain portions that contain content of interest.

[0019] One innovative aspect of the subject matter described herein is embodied in a method that includes obtaining an image captured using a camera coupled to a wearable multimedia device, comparing the image to one or more known images, identifying one or more target features in the image based on comparing the image to the one or more known images, generating a new image that includes a portion of the image that includes the one or more target features, and providing the new image to the wearable multimedia device for presentation on a display associated with the wearable multimedia device.

[0020] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs configured to perform the operations of the methods encoded on a computer storage device. A system of one or more computers may be so configured by software, firmware, hardware, or combinations thereof installed on the system that, when in operation, causes the system to perform the operations. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the device to perform the operations.

[0021] Each of the foregoing and other embodiments may optionally include one or more of the following features, alone or in combination: By way of example, in some implementations, the operations include identifying a matching image of the one or more known images that best matches the image captured using the camera based on comparing the image to one or more known images, identifying one or more features in the matching image, and identifying one or more target features in the image based on identifying the one or more features in the matching image, where at least one of the one or more target features is a feature of the one or more features in the matching image.

[0022] In some implementations, identifying a matching image from one or more known images that best matches the image captured using the camera includes comparing one or more features of the matching image to one or more target features in the image.

[0023] In some implementations, the one or more known images include a reference image specified by a user of the wearable multimedia device.

[0024] In some implementations, the one or more target features include one or more features in a reference image, and the operations include analyzing the reference image based at least on receiving a designation of the reference image by a user of the wearable multimedia device, identifying one or more features in the reference image based on the analysis, determining one or more parameters for each of the one or more features, comparing the one or more parameters to respective thresholds, and selecting the feature as the target feature when it is determined that the one or more parameters satisfy the respective thresholds.

[0025] In some implementations, the one or more parameters include a focus level.

[0026] In some implementations, the one or more target features include a representation of an object.

[0027] In some implementations, the object includes one or more of a human, an animal, or a vehicle.

[0028] In some implementations, providing and displaying the new image includes using a laser projector of the wearable multimedia device to project the new image onto a display surface that is physically separate from the wearable multimedia device.

[0029] In some implementations, the display surface includes one of the surface of a user's hand, the palm of a user's hand, a user's wrist, a table top, a wall, a portion of glass, or a portion of the ground.

[0030] In some implementations, the computing device is a server that is communicatively coupled to the wearable multimedia device.

[0031] In some implementations, the computing device is a multimedia device.

[0032] In some implementations, the actions include generating a higher resolution version of an image captured using a camera coupled to the wearable multimedia device.

[0033] In some implementations, generating the higher resolution version of the image includes providing the image to a machine learning model and generating the higher resolution version of the image based on obtained output from the machine learning model.

[0034] The details of the disclosed embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. [Brief description of the drawings]

[0035] [Figure 1] FIG. 1 is a block diagram of an operating environment for a wearable multimedia device and a cloud computing platform with an application ecosystem for processing multimedia data captured by the wearable multimedia device, according to one embodiment. [Diagram 2] 2 is a block diagram of a data processing system implemented by the cloud computing platform of FIG. 1 according to one embodiment. [Diagram 3]FIG. 2 is a block diagram of a data processing pipeline for processing a contextual data stream, according to one embodiment. [Figure 4] FIG. 13 is a block diagram of another data process for processing a contextual data stream for a transportation application, according to one embodiment. [Diagram 5] 3 illustrates a data object used by the data processing system of FIG. 2 according to one embodiment. [Figure 6] FIG. 1 is a flow diagram of a data pipeline process, according to one embodiment. [Figure 7] FIG. 1 is a diagram of a cloud computing platform architecture, according to one embodiment. [Figure 8] FIG. 2 is a diagram of the architecture of a wearable multimedia device according to one embodiment. [Figure 9] FIG. 1 is a system block diagram of a projector architecture, according to one embodiment. [Figure 10A] FIG. 1 is a diagram of a system for directed cropping of an image. [Figure 10B] FIG. 2 is a diagram of an exemplary virtual interface according to one embodiment. [Figure 11] FIG. 1 is a diagram of a system for generating a cropped image. [Figure 12] FIG. 1 is a flow diagram of a process for generating a cropped image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] The use of the same reference numbers in the various drawings indicates similar elements.

[0037] Exemplary Wearable Multimedia Device The features and processes described herein can be implemented on a wearable multimedia device. In one embodiment, the wearable multimedia device is a lightweight, small form factor, battery-powered device that can be attached to a user's clothing or an object using a tension clasp, an interlocking pin back, a magnet, or any other attachment mechanism. The wearable multimedia device includes a digital image capture device (e.g., a camera with 180° FOV and optical image stabilization (OIS)) that allows a user to capture multimedia data (e.g., video, audio, depth data, biometric data) of life events ("moments") and document transactions (e.g., financial transactions) spontaneously and / or continuously with minimal user interaction or device setup. The multimedia data captured by the wireless multimedia device ("context data") is uploaded to a cloud computing platform with an application ecosystem that allows the context data to be processed, edited, and formatted by one or more applications (e.g., artificial intelligence (AI) applications) into any desired presentation format (e.g., single image, image stream, video clip, audio clip, multimedia presentation, image gallery) that can be downloaded and replayed on the wearable multimedia device and / or any other playback device. For example, the cloud computing platform can convert the video and audio data into any desired filmmaking style (e.g., documentary, lifestyle, candid, photojournalism, sports, street) specified by the user.

[0038] In one embodiment, the context data is processed by a server computer of the cloud computing platform based on the user's preferences. For example, based on the user's preferences, the image can be color graded, stabilized, and cropped to perfectly suit the moment the user wants to re-experience. The user's preferences can be stored in a user profile created by the user through an online account accessible through a website or portal, or the user's preferences can be learned over time by the platform (e.g., using machine learning). In one embodiment, the cloud computing platform is a scalable distributed computing environment. For example, the cloud computing platform can be a distributed streaming platform (e.g., Apache Kafka™) with real-time streaming data pipelines and real-time streaming applications that transform or react to the streams of data.

[0039] In one embodiment, a user can start and stop a context data capture session on the wearable multimedia device using simple touch gestures (e.g., taps or swipes), by speaking commands, or by any other input mechanism. All or a portion of the wearable multimedia device can be automatically powered down when the wearable multimedia device detects using one or more sensors (e.g., proximity sensors, optical sensors, accelerometers, gyroscopes) that it is not being worn by a user.

[0040] The contextual data may be encrypted, compressed, and stored in an online database associated with a user account using any desired encryption or compression technique. The contextual data may be stored for a specified period of time that may be set by the user. Users may be provided with opt-in mechanisms and other tools to manage their data and data privacy through a website, portal, or mobile application.

[0041] In one embodiment, the context data includes point cloud data to provide three-dimensional (3D) surface mapped objects that can be processed using, for example, Augmented Reality (AR) and Virtual Reality (VR) applications in the application ecosystem. The point cloud data can be generated by a depth sensor (e.g., LiDAR or Time of Flight (TOF)) embedded in the wearable multimedia device.

[0042] In one embodiment, the wearable multimedia device includes a Global Navigation Satellite System (GNSS) receiver (e.g., Global Positioning System (GPS)) and one or more inertial sensors (e.g., accelerometer, gyroscope) for determining the location and orientation of the user wearing the device at the time the context data was captured. In one embodiment, one or more images in the context data can be used by a location application, such as a visual odometry application, in the application ecosystem to determine the user's position and orientation.

[0043] In one embodiment, the wearable multimedia device may also include one or more environmental sensors, including but not limited to ambient light sensors, magnetometers, pressure sensors, voice activity detectors, etc. This sensor data may be included in the context data to enhance the content presentation with additional information that may be used to capture moments.

[0044] In one embodiment, the wearable multimedia device may include one or more biometric sensors, such as a heart rate sensor, a fingerprint scanner, etc. This sensor data may be included in the context data to document a transaction or to indicate the user's emotional state during a moment (e.g., an elevated heart rate may indicate excitement or anxiety).

[0045] In one embodiment, the wearable multimedia device includes a headphone jack for connecting a headset or earphones, and one or more microphones for receiving voice commands and capturing ambient audio. In an alternative embodiment, the wearable multimedia device includes short-range communication technologies, including but not limited to Bluetooth, IEEE 802.15.4 (ZigBee™), and Near Field Communication (NFC). The short-range communication technologies can be used to wirelessly connect to a wireless headset or earphones in addition to or instead of the headphone jack, and / or the short-range communication technologies can wirelessly connect to any other external devices (e.g., computers, printers, projectors, televisions, and other wearable devices).

[0046] In one embodiment, the wearable multimedia device includes a wireless transceiver and communication protocol stacks for various communication technologies, including Wi-Fi, 3G, 4G, and 5G communication technologies. In one embodiment, the headset or earphones also include sensors (e.g., biometric sensors, inertial sensors) that provide information about the direction the user is facing, such as to provide commands using head gestures or to play spatial audio. In one embodiment, the camera direction can be controlled by head gestures so that the camera's field of view follows the user's field of view. In one embodiment, the wearable multimedia device can be embedded or attached to the user's glasses.

[0047] In one embodiment, the wearable multimedia device includes a projector (e.g., a laser projector) or other digital projection technology (e.g., Liquid Crystal on Silicon (LCoS or LCOS) technology, Digital Light Processing (DLP) technology, or Liquid Crystal Display (LCD) technology) or can be wired or wirelessly coupled to an external projector, allowing a user to replay moments on a surface such as a wall or tabletop, or on the surface of the user's hand (e.g., the palm of the user's hand). In another embodiment, the wearable multimedia device includes an output port that can be connected to a projector or other output device.

[0048] In one embodiment, the wearable multimedia capture device includes a touch surface that responds to touch gestures (e.g., tap gestures, multi-tap gestures, or swipe gestures). The wearable multimedia device may include a small display for presenting information and one or more light indicators for indicating on / off status, power status, or any other desired status.

[0049] In one embodiment, the cloud computing platform can be driven by context-based gestures (e.g., air gestures) combined with speech queries, such as a user pointing at an object in their environment and saying "What's that building?" The cloud computing platform uses the air gestures to narrow the camera's field of view to isolate the building. One or more images of the building are captured, optionally cropped (e.g., to protect privacy), and sent to the cloud computing platform, where an image recognition application can perform the image query and store or return the results to the user. Air and touch gestures can also be implemented on a projected temporary display, for example in response to user interface elements projected onto a surface.

[0050] In one embodiment, the contextual data can be encrypted on the device and on a cloud computing platform so that only the user or any authorized viewer can re-experience the moment on a connected screen (e.g., smartphone, computer, television, etc.) or as a projection on a surface. An exemplary architecture of a wearable multimedia device is described with reference to FIG.

[0051] In addition to personal life events, wearable multimedia devices also facilitate the capture of financial transactions currently handled by smartphones. Capturing everyday transactions (e.g., business transactions, microtransactions) becomes easier, faster, and smoother by using the sight assisted contextual awareness provided by wearable multimedia devices. For example, when a user is involved in a financial transaction (e.g., making a purchase), the wearable multimedia device generates data memorializing the financial transaction, including date, time, amount, digital images or video of the parties involved, audio (e.g., the user's narration describing the transaction), and environmental data (e.g., location data). The data can be included in a multimedia data stream sent to a cloud computing platform where it can be stored online and / or processed by one or more financial applications (e.g., financial management, accounting, budgeting, tax reporting, inventory, etc.).

[0052] In one embodiment, the cloud computing platform provides a graphical user interface on a website or portal that enables various third-party application developers to upload, update, and manage their applications within the application ecosystem. Some exemplary applications can include, but are not limited to, personal live broadcasting (e.g., Instagram™ Life, Snapchat™), elderly monitoring (e.g., to make sure a loved one has taken their medication), memory recall (e.g., showing your child's soccer game last week), and personal guidance (e.g., an AI-enabled personal guidance that knows the user's location and guides the user to perform actions).

[0053] In one embodiment, the wearable multimedia device includes one or more microphones and a headset. In some embodiments, the headset wire includes the microphone. In one embodiment, a digital assistant is implemented on the wearable multimedia device that responds to user queries, requests, and commands. For example, a wearable multimedia device worn by a parent captures instantaneous context data of a child's soccer game, particularly the "moment" when the child scores a goal. The user can request (e.g., using a speech command) that the platform create a video clip of the goal and store it in his or her user account. Without any further action by the user, the cloud computing platform identifies the appropriate portion of the instantaneous context data when the goal is scored (e.g., using facial recognition, visual cues, or audio cues), compiles the instantaneous context data into a video clip, and stores the video clip in a database associated with the user account.

[0054] In one embodiment, the device may include photovoltaic surface technology to preserve battery life, and inductive charging circuitry (e.g., Qi) to enable inductive charging on a charging mat and wireless over-the-air (OTA) charging.

[0055] In one embodiment, the wearable multimedia device is configured to magnetically couple or engage with a rechargeable portable battery pack. The portable battery pack includes an engagement surface having a permanent magnet (e.g., a north pole) disposed thereon, and the wearable multimedia device has a corresponding engagement surface having a permanent magnet (e.g., a south pole) disposed thereon. Any number of permanent magnets having any desired shape or size can be arranged in any desired pattern on the engagement surfaces.

[0056] The permanent magnets hold the portable battery pack and the wearable multimedia device together in an engaged configuration with clothing (e.g., the user's shirt) between them. In one embodiment, the portable battery pack and the wearable multimedia device have the same engagement surface dimensions and therefore no overhanging portions when in the engaged configuration. The user magnetically secures the wearable multimedia device to their clothing by placing the portable battery pack under their clothing and placing the wearable multimedia device on top of the portable battery pack outside their clothing, thereby causing the permanent magnets to attract each other through the clothing.

[0057] In one embodiment, the portable battery pack has a built-in wireless power transmitter that is used to wirelessly power the wearable multimedia device while in the engaged configuration using principles of resonant inductive coupling, hi one embodiment, the wearable multimedia device includes a built-in wireless power receiver that is used to receive power from the portable battery pack while in the engaged configuration.

[0058] System Overview 1 is a block diagram of an operating environment for a wearable multimedia device and a cloud computing platform with an application ecosystem for processing multimedia data captured by the wearable multimedia device, according to one embodiment. The operating environment 100 includes a wearable multimedia device 101, a cloud computing platform 102, a network 103, an application ("app") developer 104, and a third-party platform 105. The cloud computing platform 102 is coupled to one or more databases 106 for storing contextual data uploaded by the wearable multimedia device 101.

[0059] As previously described, the wearable multimedia device 101 is a lightweight, small form factor, battery-powered device that can be attached to a user's clothing or an object using a tension clasp, interlocking pinbacks, magnets, or any other attachment mechanism. The wearable multimedia device 101 includes a digital image capture device (e.g., a camera with 180° FOV and OIS) that allows a user to spontaneously capture "instant" multimedia data (e.g., video, audio, depth data) and document daily transactions (e.g., financial transactions) with minimal user interaction or device setup. The contextual data captured by the wireless multimedia device 101 is uploaded to the cloud computing platform 102. The cloud computing platform 102 includes an application ecosystem that allows the contextual data to be processed, edited, and formatted by one or more server-side applications into any desired presentation format (e.g., single image, image stream, video clip, audio clip, multimedia presentation, image gallery) that can be downloaded and replayed on the wearable multimedia device and / or other playback devices.

[0060] As an example, at a child's birthday party, a parent can clip the wearable multimedia device to their clothing (or attach the device to a necklace or chain and wear it around their neck) with the camera lens facing their line of sight. The camera includes a 180° FOV, which allows the camera to capture almost everything the user is currently looking at. The user can simply tap the surface of the device, or press a button, or speak a command to start recording. No additional setup is required. A multimedia data stream (e.g., video with audio) is recorded capturing special moments of the birthday (e.g., blowing out the candles). This "context data" is sent in real time to the cloud computing platform 102 over a wireless network (e.g., Wi-Fi, cellular). In one embodiment, the context data is stored on the wearable multimedia device so that it can be uploaded at a later time. In another embodiment, the user can transfer the context data to another device (e.g., a personal computer hard drive, a smartphone, a tablet computer, a thumb drive) and later use an application to upload the context data to the cloud computing platform 102.

[0061] In one embodiment, the contextual data is processed by one or more applications of an application ecosystem hosted and managed by the cloud computing platform 102. The applications can be accessed through their respective application programming interfaces (APIs). A custom distributed streaming pipeline is created by the cloud computing platform 102 to process the contextual data based on one or more of data type, data volume, data quality, user preferences, templates, and / or any other information to generate a desired presentation based on the user preferences. In one embodiment, machine learning techniques can be used to automatically select appropriate applications for inclusion in the data processing pipeline, with or without user preferences. For example, historical user context data stored in a database (e.g., a NoSQL database) can be used to determine the user preferences for data processing using any suitable machine learning techniques (e.g., deep learning neural networks or convolutional neural networks).

[0062] In one embodiment, the application ecosystem can include a third-party platform 105 that processes the context data. A secure session is set up between the cloud computing platform 102 and the third-party platform 105 to send / receive the context data. This design allows the third-party app provider to control access to their application and provide updates. In another embodiment, the application runs on the cloud computing platform 102's servers and updates are sent to the cloud computing platform 102. In the latter embodiment, the app developer 104 can use APIs provided by the cloud computing platform 102 to upload and update applications to be included in the application ecosystem.

[0063] Exemplary Data Processing System Figure 2 is a block diagram of a data processing system implemented by the cloud computing platform of Figure 1, according to one embodiment. Data processing system 200 includes a recorder 201, a video buffer 202, an audio buffer 203, a photo buffer 204, a capture server 205, data storage 206, a video processor 207, an audio processor 208, a photo processor 209, and a third party processor 210.

[0064] A recorder 201 (e.g., a software application) running on the wearable multimedia device records the video, audio, and photo data ("context data") captured by the camera and audio subsystems and stores them in buffers 202, 203, 204, respectively. This context data is then sent (e.g., using wireless OTA technology) to an ingestion server 205 of the cloud computing platform 102. In one embodiment, the data can be sent in separate data streams, each with a unique stream identifier (streamid). A stream is a separate piece of data that can contain the following exemplary attributes: location (e.g., latitude, longitude), user, audio data, a video stream of variable duration, and N photos. A stream can have a duration from 1 to MAXSTREAM_LEN seconds, where in this example MAXSTREAM_LEN=20 seconds.

[0065] The ingestion server 205 captures the streams and creates stream records in the data store 206 to store the results of the processors 207-209. In one embodiment, the audio stream is processed first and used to determine other streams that are required. The ingestion server 205 sends the streams to the appropriate processors 207-209 based on the streamid. For example, the video stream is sent to the video processor 207, the audio stream is sent to the audio processor 208, and the photo stream is sent to the photo processor 209. In one embodiment, at least a portion of the data (e.g., image data) collected from the wearable multimedia device is processed into metadata and encrypted so that it can be further processed by a given application and sent back to the wearable multimedia device or other devices.

[0066] The processors 207-209 may execute proprietary or third-party applications as previously described. For example, the video processor 207 may be a video processing server that sends raw video data stored in the video buffer 202 to a set of one or more image processing / editing applications 211, 212 based on user preferences or other information. The processor 207 sends requests to the applications 211, 212 and returns results to the ingest server 205. In one embodiment, the third-party processor 210 may process one or more of the streams using its own processor and application. In another example, the audio processor 208 may be an audio processing server that sends speech data stored in the audio buffer 203 to a speech-to-text application 213.

[0067] Exemplary Scene Identification Application 3 is a block diagram of a data processing pipeline for processing a contextual data stream, according to one embodiment. In this embodiment, a data processing pipeline 300 is created and configured to determine what a user sees based on contextual data captured by a wearable multimedia device worn by the user. An ingestion server 301 receives an audio stream (e.g., including a user's narration) from an audio buffer 203 of the wearable multimedia device and sends the audio stream to an audio processor 305. The audio processor 305 sends the audio stream to an app 306, which performs speech-to-text conversion and returns the parsed text to the audio processor 305. The audio processor 305 returns the parsed text to the ingestion server 301.

[0068] The video processor 302 receives the parsed text from the capture server 301 and sends a request to the video processing app 307. The video processing app 307 identifies objects in the video scene and labels the objects using the parsed text. The video processing app 307 sends a response describing the scene (e.g., labeled objects) to the video processor 302. The video processor then forwards the response to the capture server 301. The capture server 301 sends the response to the data merge process 308, which merges the response with the user's location, direction, and map data. The data merge process 308 returns a response with the scene description to the recorder 304 on the wearable multimedia device. For example, the response can include text describing the scene as a child's birthday party, including a map location and a description of the objects in the scene (e.g., identifying people in the scene). The recorder 304 associates the scene description with the multimedia data stored on the wearable multimedia device (e.g., using the streamid). When the user recalls the data, the data is enriched with the scene description.

[0069] In one embodiment, the data merge process 308 can use more than just location and map data. There may be concepts of ontologies. For example, facial features of a user's father captured in an image may be recognized by a cloud computing platform and returned as "Dad" rather than the user's name, and an address such as "555 Main Street, San Francisco, CA" may be returned as "Home." Ontologies can be specific to a user and can grow and learn from the user's input.

[0070] Example Transportation Application FIG. 4 is another data processing block diagram for processing a contextual data stream for a transportation application, according to one embodiment. In this embodiment, a data processing pipeline 400 is created to call a transportation company (e.g., Uber®, Lyft®) for a ride home. Contextual data from the wearable multimedia device is received by an ingestion server 401, and an audio stream from the audio buffer 203 is sent to an audio processor 405. The audio processor 405 sends the audio stream to an app 406, which converts the speech to text. The parsed text is returned to the audio processor 405, which returns the parsed text (e.g., a user speech request for transportation) to the ingestion server 401. The processed text is sent to a third party processor 402. The third party processor 402 sends the user location and a token to a third party application 407 (e.g., an Uber® application or a Lyft™® application). In one embodiment, the token is an API and authorization token used to mediate requests on behalf of the user. The application 407 returns a response data structure to the third party processor 402, which forwards the response data structure to the ingestion server 401. The ingestion server 401 checks the ride arrival status (e.g., ETA) in the response data structure and sets up a callback to the user in the user callback queue 408. The ingestion server 401 returns a response with a vehicle description to the recorder 404, which the digital assistant can speak to the user through a loudspeaker on the wearable multimedia device or through the user's headphones or earphones via a wired or wireless connection.

[0071] FIG. 5 illustrates data objects used by the data processing system of FIG. 2 according to one embodiment. The data objects are part of a software component infrastructure instantiated on a cloud computing platform. The “Streams” object includes the following data: streamid, deviceid, start, end, lat, lon, attributes, and entities. “streamid” identifies the stream (e.g., video, audio, photo), “deviceid” identifies the wearable multimedia device (e.g., mobile device ID), “start” is the start time of the context data stream, “end” is the end time of the context data stream, “lat” is the latitude of the wearable multimedia device, “lon” is the longitude of the wearable multimedia device, “attributes” include, e.g., birth date, facial point, skin tone, audio characteristics, address, phone number, etc., and “entities” constitute the ontology. For example, the name “John Do” is mapped to “Dad” or “Brother” depending on the user.

[0072] The "Users" object includes the following data: userid, deviceid, email, fname, and lname. userid identifies the user with a unique identifier, deviceid identifies the wearable device with a unique identifier, email is the user's registered email address, fname is the user's first name, and lname is the user's last name. The "Userdevices" object includes the following data: userid and deviceid. The "Devices" object includes the following data: deviceid, started, state, modified, and created. In one embodiment, deviceid is a unique identifier for the device (different from, for example, a MAC address). started is when the device was first started. state is on / off / sleep. modified is the last update date reflecting the last state change or operating system (OS) change. created is when the device was turned on for the first time.

[0073] The "ProcessingResults" object contains the following data: streamid, ai, result, callback, duration, and accuracy. In one embodiment, streamid is each user stream as a universally unique identifier (UUID). For example, a stream that started between 8:00AM and 10:00AM will have id:15h158dhb4, and a stream that starts between 10:15AM and 10:18AM will have the UUID that was contacted for this stream. ai is the identifier of the platform application that was contacted for this stream. result is the data sent from the platform application. callback is the callback that was used (the callback is tracked in case the version changes and thus the platform needs to replay the request). accuracy is a score on how accurate the result set is. In one embodiment, the processing results can be used for multiple tasks, such as 1) notifying the merge server of the set of results, 2) determining the fastest ai so that the user experience can be improved, and 3) determining the most accurate ai. Depending on the use case, speed may be preferred over accuracy, or vice versa.

[0074] An "Entities" object contains the following data: entityID, userID, entityName, entityType, and entityAttribute. entityID is the UUID of the entity, and if an entity has multiple entries, the entityID refers to one of the entities. For example, "Barack Obama" would have an entityID of 144, which can be linked in an association table to POTUS44, or "Barack Hussein Obama", or "President Obama". userID identifies the user who created the entity record. entityName is what the userID calls the entity. For example, the entityName for Malia Obama's entityID 144 could be "Dad" or "Papa". entityType is a person, place, or thing. entityAttributes are a number of attributes about the entity, which are specific to the userID's understanding of that entity. This allows entities to map one to the other, so, for example, when Malia utters a speech query like "Can you see your dad?", the cloud computing platform can translate that query to Barack Hussein Obama and use that when brokering requests to third parties or looking up information within the system.

[0075] Example Process 6 is a flow diagram of a data pipeline process according to one embodiment. The process 600 can be implemented using the wearable multimedia device 101 and cloud computing platform 102 described with reference to FIGS. 1-5.

[0076] Process 600 can begin with receiving contextual data from a wearable multimedia device (601). For example, the contextual data can include video, audio, and still images captured by the camera and audio subsystem of the wearable multimedia device.

[0077] The process 600 can continue by creating (e.g., instantiating) 602 a data processing pipeline with applications based on the context data and the user's requirements / preferences. For example, based on the user's requirements or preferences and also based on the data type (e.g., audio, video, photo), one or more applications can be logically connected to form a data processing pipeline for processing the context data into a presentation that is played on the wearable multimedia device or another device.

[0078] The process 600 can continue by processing 603 the contextual data in a data processing pipeline. For example, speech from a user's narration during a moment or transaction can be converted to text, which is then used to label objects in a video clip.

[0079] The process 600 may continue by sending (604) the output of the data processing pipeline to a wearable multimedia device and / or other playback device.

[0080] Exemplary Architecture of a Cloud Computing Platform FIG. 7 is an exemplary architecture 700 of the cloud computing platform 102 described with reference to FIGS. 1-6, according to one embodiment. Other architectures are possible, including architectures with more or fewer components. In some implementations, the architecture 700 includes one or more processors 702 (e.g., a dual-core Intel® Xeon® processor), one or more network interfaces 706, one or more storage devices 704 (e.g., hard disks, optical disks, flash memory), and one or more computer-readable media 708 (e.g., hard disks, optical disks, flash memory, etc.). These components can communicate and exchange data via one or more communication channels 710 (e.g., a bus), which can utilize various hardware and software to facilitate the transfer of data and control signals between the components.

[0081] The term "computer-readable medium" refers to any medium that participates in providing instructions to the processor 702 for execution, including, but not limited to, non-volatile media (e.g., optical or magnetic disks), volatile media (e.g., memory), and transmission media. Transmission media include, but are not limited to, coaxial cables, copper wire, and fiber optics.

[0082] The computer readable medium 708 may further include an operating system 712 (eg, Mac OS Server, Windows NT Server, Linux Server), a network communication module 714 , data processing instructions 716 , and interface instructions 718 .

[0083] The operating system 712 may be multi-user, multi-processing, multi-tasking, multi-threading, real-time, etc. The operating system 712 performs basic tasks including, but not limited to, recognizing input from and providing output to the devices 702, 704, 706, and 708, keeping track of and managing files and directories on the computer-readable medium 708 (e.g., memory or storage device), controlling peripheral devices, and managing traffic on one or more communication channels 710. The network communication module 714 includes various components for establishing and maintaining network connections (e.g., software for implementing communication protocols such as TCP / IP, HTTP, etc.), and various components for creating a distributed streaming platform using, for example, Apache Kafka™. The data processing instructions 716 include server-side or back-end software for implementing the server-side operations described with reference to Figures 1-6. The interface instructions 718 include software for implementing a web server and / or portal for sending data to and receiving data from the wearable multimedia device 101, third party application developers 104, and third party platforms 105 described with reference to FIG. 1.

[0084] The architecture 700 can be included in any computing device, including one or more server computers in a local or distributed network, each having one or more processing cores. The architecture 700 can be implemented in a parallel processing infrastructure or a peer-to-peer infrastructure, or on a single device with one or more processors. The software can include multiple software components or can be a single body of code.

[0085] Exemplary Architecture of a Wearable Multimedia Device Figure 8 is a block diagram of an example architecture 800 of a wearable multimedia device implementing the features and processes described with reference to Figures 1-6. The architecture 800 may include a memory interface 802, a data processor, image processor, or central processing unit 804, and a peripheral interface 806. The memory interface 802, the processor 804, or the peripheral interface 806 may be separate components or may be integrated in one or more integrated circuits. One or more communication buses or signal lines may couple the various components.

[0086] Sensors, devices, and subsystems can be coupled to the peripherals interface 806 to facilitate multiple functions. For example, a motion sensor 810, a biometric sensor 812, and a depth sensor 814 can be coupled to the peripherals interface 806 to facilitate motion, orientation, biometric, and depth detection functions. In some implementations, motion sensors 810 (e.g., accelerometers, rate gyroscopes) can be utilized to detect movement and orientation of the wearable multimedia device.

[0087] Other sensors, such as environmental sensors (e.g., temperature sensors, barometers, ambient light), can also be connected to the peripherals interface 806 to facilitate environmental sensing functions. For example, biometric sensors can detect fingerprints, facial recognition, heart rate, and other fitness parameters. In one embodiment, a haptic motor (not shown) can be coupled to the peripherals interface and can provide vibration patterns as haptic feedback to the user.

[0088] To enable georeferencing, a location processor 815 (e.g., a GNSS receiver chip) may be connected to the peripherals interface 806. An electronic magnetometer 816 (e.g., an integrated circuit chip) may also be connected to the peripherals interface 806 to provide data that may be used to determine the direction of magnetic north. Thus, the electronic magnetometer 816 may be used by an electronic compass application.

[0089] A camera subsystem 820 and an optical sensor 822, such as a charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) optical sensor, may be utilized to facilitate camera functions such as recording of photographs and video clips. In one embodiment, the camera has a 180° FOV and OIS. The depth sensor may include an infrared emitter that projects a known pattern of dots onto the object / subject. The dots are then photographed by a dedicated infrared camera and analyzed to determine depth data. In one embodiment, a time-of-flight (TOF) camera may be used to determine distance based on the known speed of light and measuring the time of flight of the light signal between the camera and the object / subject for each point in the image.

[0090] The communication functions may be facilitated through one or more communication subsystems 824. The communication subsystem 824 may include one or more wireless communication subsystems. The wireless communication subsystem 824 may include radio frequency receivers and transmitters and / or optical (e.g., infrared) receivers and transmitters. The wired communication system may include a port device, such as a Universal Serial Bus (USB) port or some other wired port connection, that may be used to establish a wired connection to other computing devices, such as other communication devices, network access devices, personal computers, printers, display screens, or other processing devices capable of receiving or transmitting data (e.g., projectors).

[0091] The specific design and implementation of the communications subsystem 824 may depend on the communications network or medium over which the device is intended to operate. For example, the device may include wireless communications subsystems designed to operate over Global System for Mobile Communications (GSM) networks, GPRS networks, Enhanced Data GSM Environment (EDGE) networks, IEEE 802.xx communications networks (e.g., Wi-Fi, WiMax, ZigBee™), 3G networks, 4G networks, 4G LTE networks, Code Division Multiple Access (CDMA) networks, Near Field Communication (NFC), Wi-Fi Direct, and Bluetooth™ networks. The wireless communications subsystem 824 may include a hosting protocol such that the device may be configured as a base station for other wireless devices. As another example, the communications subsystem may enable the device to synchronize with a host device using one or more protocols or communications technologies, such as, for example, the TCP / IP protocol, the HTTP protocol, the UDP protocol, the ICMP protocol, the POP protocol, the FTP protocol, the IMAP protocol, the DCOM protocol, the DDE protocol, the SOAP protocol, HTTP Live Streaming, MPEG Dash, and any other known communications protocol or technology.

[0092] An audio subsystem 826 may be coupled to a speaker 828 and one or more microphones 830 to facilitate voice-enabled functions such as voice recognition, voice duplication, digital recording, telephony, and beamforming.

[0093] The I / O subsystem 840 can include a touch controller 842 and / or another input controller 844. The touch controller 842 can be coupled to a touch surface 846. The touch surface 846 and touch controller 842 can detect contact and movement or their interruption using any of a number of touch sensitivity technologies including, for example, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements to determine one or more points of contact with the touch surface 846. In one implementation, the touch surface 846 can display virtual or soft buttons, which a user can use as an input / output device.

[0094] Other input controllers 844 may be coupled to other input / control devices 848, such as one or more buttons, rocker switches, thumb wheels, infrared ports, USB ports, and / or a pointer device such as a stylus. The one or more buttons (not shown) may include up / down buttons for adjusting the volume of the speaker 828 and / or microphone 830.

[0095] Additionally, a projector subsystem 832 can be connected to the peripherals interface 806 to visually present information to the user in the form of projected light. For example, the projector subsystem 832 can project light onto a surface according to a specific spatial and / or temporal pattern such that the user perceives text, images, videos, colors, patterns, and / or any other graphical information on the surface. In some implementations, the projector subsystem 832 can project light onto a surface of the user's body, such as the user's hand or palm. In some implementations, the projector subsystem 832 can project light onto a surface other than the user's body, such as a wall, table, desk, or any other object. The projector subsystem 832 is described in more detail with reference to FIG. 9.

[0096] In some implementations, the projector subsystem 832 projects light onto a surface to provide interactive VIs to a user. For example, the projector subsystem 832 can project light onto a surface so that the user perceives one or more interactive user interface elements (e.g., selectable buttons, dials, switches, boxes, images, videos, text, icons, etc.). Furthermore, the user can interact with the VIs by performing one or more gestures on the VIs and user interface elements. For example, the user can perform a pointing gesture, a tap gesture, a swipe gesture, a wave gesture, or any other gesture using his or her hand and / or fingers. The wearable multimedia device can detect the performed gestures using one or more sensors (e.g., the camera / video subsystem 820, the environmental sensor 817, the depth sensor 814, etc.), identify one or more commands associated with the gestures, and execute the identified commands (e.g., using the processor 804). Exemplary VIs are described in further detail below.

[0097] In some implementations, device 800 plays recorded audio and / or video files (including spatial audio) to a user, such as MP3 files, AAC files, spatial audio files, and MPEG video files. In some implementations, device 800 can include MP3 player functionality and can include pin connectors or other ports for interfacing with other devices. Other input / output and control devices can also be used. In one embodiment, device 800 can include an audio processing unit for streaming audio to an accessory device via a direct or indirect communication link.

[0098] The memory interface 802 can be coupled to a memory 850. The memory 850 can include high-speed random access memory or non-volatile memory such as one or more magnetic disk storage devices, one or more optical storage devices, or flash memory (e.g., NAND, NOR). The memory 850 can store an operating system 852, such as Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or an embedded operating system such as VxWorks. The operating system 852 can include instructions for handling basic system services and for performing hardware-dependent tasks. In some implementations, the operating system 852 can include a kernel (e.g., a UNIX kernel).

[0099] The memory 850 may also store communications instructions 854 for facilitating communications with one or more additional devices, one or more computers or servers, including peer-to-peer communications with wireless accessory devices, as described with reference to Figures 1-6. The communications instructions 854 may also be used to select an operating mode or communications medium for use by the device based on the geographic location of the device.

[0100] Memory 850 may include sensor processing instructions 858 for facilitating sensor-related processing and functions, and recorder instructions 860 for facilitating recording functions, as described with reference to Figures 1-6. Other instructions may include GNSS / navigation instructions for facilitating GNSS and navigation-related processes, camera instructions for facilitating camera-related processes, and user interface instructions for facilitating user interface processing, including touch models for interpreting touch input.

[0101] Each of the above-identified instructions and applications may correspond to a set of instructions for implementing one or more functions described above. These instructions need not be implemented as separate software programs, procedures, or modules. Memory 850 may include additional instructions or fewer instructions. Furthermore, various functions of the device may be implemented in hardware and / or software, including one or more signal processing circuits and / or application specific integrated circuits (ASICs).

[0102] 9 is a system block diagram of a projector subsystem 832, according to one embodiment. The projector subsystem 832 scans pixels in two dimensions, images a 2D array of pixels, or mixes imaging and scanning. Scanning projectors directly utilize the narrow divergence and two-dimensional (2D) scanning of a laser beam to "paint" an image pixel by pixel. In some embodiments, separate scanners are used for the horizontal and vertical scan directions. In other embodiments, a single two-axis scanner is used. The specific beam trajectory also varies depending on the type of scanner used.

[0103] In the illustrated example, the projector subsystem 832 is a scanning picoprojector that includes a controller 901, a battery 902, a power management chip (PMIC) 903, a solid-state laser 904, an XY scanner 905, a driver 906, a memory 907, a digital-to-analog converter (DAC) 908, and an analog-to-digital converter (ADC) 909.

[0104] The controller 901 provides control signals to the XY scanner 905. In response to the control signals, the XY scanner 905 uses movable mirrors to steer the laser beam generated by the solid-state laser 904 in two dimensions. The XY scanner 905 includes one or more microelectromechanical (MEMS) micromirrors with tilt angles controllable in one or two dimensions. The driver 906 includes power amplifiers and other electronic circuits (e.g., filters, switches) to provide control signals (e.g., voltage or current) to the XY scanner 905. The memory 907 stores various data used by the projector, including laser patterns for projected text and images. The DAC 908 and ADC 909 perform data conversion between the digital and analog domains. The PMIC 903 manages the power and duty cycle of the solid-state laser 904, including turning it on and off and adjusting the amount of power provided to the solid-state laser 904. The solid-state laser 904 can be, for example, a vertical cavity surface emitting laser (VCSEL).

[0105] In one embodiment, the controller 901 uses image data from the camera / video subsystem 820 and / or depth data from the depth sensor 814 to recognize and track the position of the user's hands and / or fingers on the laser projection so that user input is received by the wearable multimedia device 101 using the laser projection as an input interface.

[0106] In another embodiment, the projector subsystem 832 uses a vector graphic projection display and low power fixed MEMS micromirrors to conserve power. The projector subsystem 832 includes a depth sensor so that the projected area can be masked as needed to avoid projecting onto a finger / hand that is interacting with the laser projected image. In one embodiment, the depth sensor can also track gestures to control input on another device (e.g., swiping an image on a TV screen, interacting with a computer, a smart speaker, etc.).

[0107] In other embodiments, liquid crystal on silicon (LCoS or LCOS) digital projection technology, digital light processing (DLP) digital projection technology, or liquid crystal display (LCD) digital projection technology can be used instead of the picoprojector.

[0108] Example Virtual Interface As described above, the wearable multimedia device 101 can include a projector subsystem 832 configured to visually present information to a user in the form of projected light. For example, the projector subsystem 832 can project light onto a surface (e.g., the surface of a user's hand, such as the palm of the user's hand) according to a particular spatial and / or temporal pattern such that the user perceives a VI with one or more user interface elements. In some implementations, the VI and / or user interface elements can include any combination of text, images, videos, colors, patterns, shapes, lines, or any other graphical information.

[0109] Further, the user can perform gestures to interact with the VI. By way of example, the user can perform one or more gestures directed at one or more of the user interface elements. By way of example, the user can point at a user interface element, touch or tap a user interface element (e.g., once or multiple times in succession) with his / her finger, perform a swiping motion along a user interface element with his / her finger, wave at a user interface element with his / her hand, hover over a user interface element, or perform any other gesture with a hand or finger. The wearable multimedia device 101 can detect the performed gestures using one or more sensors (e.g., the camera / video subsystem 820, the environmental sensor 817, the depth sensor 814, etc.), identify one or more commands associated with the gestures, and execute the identified commands (e.g., using the processor 804).

[0110] At least some of the user interface elements and / or commands can be used to control the operation of the wearable multimedia device 101. For example, at least some of the user interface elements and / or commands can be used to perform or control the generation of video and / or audio content, viewing of content, editing of content, storage and transmission of data, and / or any other operation described herein. In some implementations, the wearable multimedia device 101 can present a VI that allows the user to view images or interact with the displayed images, for example, by marking the displayed images as favorites or sharing the displayed images with one or more other users. This VI can also be referred to as a camera interface. An example of a camera interface is shown in FIG. 10B, which shows a camera interface 1028 projected by the projector subsystem 832 onto the palm 1026 of the user's hand. The camera interface 1028 displays a representation of the cropped image 1018, which is generated as described in more detail below with reference to FIGS. 10A-10B, 11, and 12.

[0111] 10A illustrates an example of a system 1000 for directed cropping of an image. The system 1000 includes a device wearable multimedia device 101 with an image capture device 1002 component, and a network device 1016. In some implementations, the image capture device 1002 corresponds to the camera subsystem 820.

[0112] In some implementations, the network device 1016 is a network server communicatively coupled to the wearable device 101. For example, in some implementations, the network device 1016 is on a network server in the cloud computing platform 102. In some implementations, the network device 1016 includes one or more processors that perform the directed trimming described below. In some cases, the one or more processors are similar to the processor 702.

[0113] The following description is with respect to the network device 1016 being separate from the wearable multimedia device 101, although in some implementations, as indicated above, the wearable multimedia device 101 includes one or more processors for performing the directed trimming operations detailed below. In some implementations, the wearable multimedia device 101 and the network device 1016 cooperate to perform the directed trimming operations.

[0114] In some implementations, an image capture device 1002 is used to capture one or more images of a scene, and the image data is provided to a network device 1016. For example, the wearable multimedia device 101 can provide image data, such as an image 1004 captured using the image capture device 1002, to the network device 1016. The network device 1016 processes the image 1004 using its one or more processors.

[0115] In some implementations, image 1004 is a wide field of view image that represents a wide field of view scene. For example, as shown, image 1004 includes object 1006, people 1008, 1010, 1012, and 1014. Person 1008 is not looking at image capture device 1002 when image 1004 is captured. People 1010, 1012, and 1014 are looking at image capture device 1002 when image 1004 is captured.

[0116] In some implementations, the network device 1016 includes one or more trained models. For example, the model can be trained using a set of one or more images. The one or more images can be selected by a user or by an automated process, such as one or more algorithms or trained models.

[0117] In some implementations, the network device 1016 executes one or more trained models using one or more component processors to extract one or more features from the image 1004. For example, the trained model may extract features including one or more of the people 1010, 1012, and 1014 in the focus frame of the image looking at the image capture device 1002. The trained model may extract another feature including a person 1008 outside the focus frame who is not looking at the image capture device 1002. The model trained by the network device 1016 may also extract features indicative of unrelated objects in the image, such as background objects, e.g., object 1006.

[0118] In some implementations, the network device 1016 provides network device 1016 data corresponding to the image 1004 to a model trained to increase the resolution of the image 1004. The model can generate one or more additional pixels based on pixels of the image 1004. In some implementations, the model interpolates pixel values ​​between two pixel values ​​based on portions of pixels that include the two pixel values.

[0119] The network device 1016 generates a cropped image 1018. The cropped image 1018 includes the people 1010, 1012, and 1014. The cropped image 1018 is a cropped version of the image 1004. In some implementations, the portion of the image 1004 used for the cropped image 1018 is determined using one or more trained models. In some implementations, the portion of the image 1004 used for the cropped image 1018 is determined using one or more algorithms. In some implementations, the network device 1016 identifies areas of the image 1004 that are in focus, areas that include an object of interest, areas that do not include out-of-focus subjects or objects, or areas that do not include extraneous objects. For example, as discussed in FIG. 10B, the network device 1016 can generate one or more features from one or more reference images. The features may indicate, among other things, one or more subjects or objects that should be kept in focus, subjects or objects of interest, and subjects or objects of non-interest, such as extraneous objects.

[0120] The network device 1016 provides the cropped image 1018 to the wearable device 101. As shown in FIG. 10B, the wearable device 101 is configured to display the cropped image 1018. As an illustrative example, FIG. 10B shows a camera interface 1028 projected by the projector subsystem 832 onto the palm of a user's hand 1026. A user can interact with the camera interface 1028 to browse digital content, select digital content, or control a digital image capture device of the wearable multimedia device 101. In the example of FIG. 10B, the camera interface 1028 includes a representation of the cropped image 1018 generated by the network device 1016.

[0121] In some implementations, the network device 1016 provides the data to one or more other devices. For example, the network device 1016 can provide the cropped image 1018 to a storage database on the network. The network device 1016 can provide the cropped image 1018 to one or more user devices, such as smartphones, televisions, monitors, among others, in addition to or instead of the wearable multimedia device 101.

[0122] In some implementations, the camera interface 1028 is configured to receive user input. For example, a user of the wearable multimedia device 101 can interact with the camera interface 1028 using a finger, or a pointing device, or other suitable means. In doing so, the user can interact with the cropped image 1018 being shown using the camera interface 1028. For example, the user can provide an input to select the cropped image 1018 as a favorite image. Alternatively, the user can provide an input indicating that the cropped image 1018 is not suitable or includes low quality characteristics such as blurry, improper cropping, among others.

[0123] In some implementations, the user input is for adjusting one or more trained models that generate the cropped image 1018. For example, the wearable device 101 can provide data to the network device 1016 indicative of a user input, such as an input from a user indicating that the cropped image 1018 is good, bad, a favorite, shared with one or more other users, or deleted, among other things. The network device 1016 can use this input as a training data point. For example, the network device 1016 can adjust one or more weights of the one or more models such that data corresponding to the image 1004 used to generate the cropped image 1018 generates a different cropped image.

[0124] In some implementations, the wearable device 101 obtains input from a user and uses the input to adjust one or more algorithms that generate the cropped image 1018. For example, the network device 1016 can determine that one or more features of the cropped image 1018 are unlikely to result in an acceptable image. In some implementations, the network device 1016 can adjust one or more weights in the summation. For example, weights corresponding to features of the cropped image 1018 can be adjusted based on user feedback on the cropped image 1018 indicating that one or more features are unlikely or likely to result in an acceptable image.

[0125] In some implementations, based on the network device 1016 receiving negative input from a user, such as a delete, a thumbs down, or other input, the weight corresponding to the feature is adjusted, e.g., upwards or downwards depending on the implementation, so that the feature is less likely to appear in the cropped image. In some implementations, based on the network device 1016 receiving positive input from a user, such as a share to another user, a like, a favorite, a viewing time period, or other input, the weight corresponding to the feature is adjusted, e.g., upwards or downwards depending on the implementation, so that the feature is more likely to appear in the cropped image.

[0126] 11 illustrates an example of a system 1100 for generating a cropped image. System 1100 illustrates one particular implementation of system 1000. System 1100 includes an image capture device 1002 and a network device 1016. In the example of FIG. 11, network device 1016 includes a cropping engine 1106 and a feature engine 1110.

[0127] As described with reference to FIG. 10A, the image capture device 1002 provides the image 1004 to the network device 1016. In some implementations, the network device 1016 provides data representing the image 1004 to the trained model 1102. For example, the trained model 1102 can be trained to generate an upscaled version 1104 of the image 1004. The model 1102 can be trained using one or more training images. The training images can include corresponding higher resolution images and lower resolution images used as ground truth data. The model 1102 can be trained to generate predictions of the higher resolution images based on the lower resolution images. The network device 1016 or another processor can train the model 1102 by comparing predicted higher resolution images from the output of the model 1102 with corresponding known high resolution images used to obtain the lower resolution images. The network device 1016 can adjust one or more weights or parameters of the model 1102 such that over time the model 1102 produces predicted higher resolution images that more closely match known higher resolution images.

[0128] The cropping engine 1106 obtains data corresponding to the image 1004. In some implementations, the cropping engine 1106 obtains an upscaled version 1104 of the image 1004. For example, the network device 1016 can provide the data corresponding to the image 1004 to the model 1102 and then obtain the output of the model 1102. The output of the model 1102 can include an upscaled version 1104 of the image 1004. The upscaled version can be useful to allow cropping to a small area without incurring a loss of resolution. Without upscaling, cropping to a small area can result in a pixelated image that is generally not pleasing to users.

[0129] In some implementations, the cropping engine 1106 provides data corresponding to the image 1004 to the trained model 1108. For example, the cropping engine 1106 can provide the upscaled image 1104 to the trained model 1108. In some implementations, the trained model 1108 uses one or more other items of data to determine which portions of the image 1104 should be cropped. For example, the cropping engine 1106 can obtain data corresponding to the target image features 1116 generated by the feature engine 1110. The trained model 1108 can identify portions of the image 1104 that include one or more of the target image features 1116. The trained model 1108 can crop the image 1104 to include one or more features in the identified portions.

[0130] The cropping engine 1106 obtains data corresponding to target image features 1116 from a feature engine 1110. In some implementations, the feature engine 1110 includes a target image 1112. In some implementations, the feature engine 1110 includes a trained model 1114. For example, the feature engine 1110 can train the model 1114 to identify abstracted features from the target image 1112 based on the target image 1112. The abstracted features can include one or more data values, such as vectors, indicative of visual features. In some implementations, the features include attention of a person in the image. In some implementations, the features include focus or resolution of a person or object in the image. In some implementations, the features include placement of a person or object. In some implementations, the features include objects of interest, such as a painting, a chandelier, a birthday cake, an athlete on a field, an athlete on a track, or an athlete in motion, among others.

[0131] In some implementations, the target image 1112 includes one or more images selected by a user. For example, the user can view a gallery of one or more images and select one or more images. In some implementations, the user views one or more images using the camera interface 1028. In general, the user can select images that he or she likes. The user may like a particular style of image, such as images that prominently feature a person or a particular person or people, images that place objects in a particular way within the frame, images with certain lighting or effects, among other things. The feature engine 1110 can take the favorited images, or liked images, or selected images, and add them to the target image 1112.

[0132] In some implementations, the model 1114 is trained to generate the target image features 1116. For example, the feature engine 1110 can provide a portion of the target image 1112 to the model 1114. The model 1114 can output one or more data values ​​indicative of one or more features. In some implementations, the feature engine 1110 tests the features predicted by the model 1114 using another portion of the target image 1112. For example, the feature engine 1110 can provide the other portion of the target image 1112 to the model 1114. The model 1114 can predict one or more features from the other portion of the target image 1112. The feature engine 1110 can compare the features predicted for the other portion to the features predicted for the first portion of the target image 1112. The feature engine 1110 can determine to add additional features to the target image features 1116 based on the comparison.

[0133] In some implementations, the feature engine 1110 updates the target image features 1116 to include one or more features identified in one or more of the target images 1112. Generally, the features may include one or more characteristics, such as a particular person, color, object placement, lighting, or other features represented in the image.

[0134] In some implementations, the network device 1016 compares data corresponding to the image 1004 with data of one or more target images, such as the target image 1112. For example, the network device 1016 can compare the image 1004 or the upscaled image 1104 with one or more target images. The one or more target images can include previously acquired images. The previously acquired images can include images from one or more user libraries. The previously acquired images can be stored on the network device 1016 or in a storage device communicatively connected to the network device 1016. In some implementations, the previously acquired images are stored on the wearable device 101.

[0135] In some implementations, one or more target images include at least one similar image. For example, image 1004 and the similar image can be of the same category. The categories can include party images, human faces, full human bodies, human portraits, natural landscapes, natural objects, and animal images, among others. Images within a category can include one or more characteristics in common with each other. For example, in at least one implementation, images in the party images category can include people in close proximity to each other. In at least one implementation, images in the human faces category can include human faces identified by one or more facial features, such as noses, eyes, and mouths, among others.

[0136] In general, the network device 1016 can determine one or more categories of a particular type based on a set of one or more common features between two or more images. The new image can be added to an existing category or can be used to generate a new category. In some implementations, a new image is combined with one or more other images to form a new category based on the one or more images sharing more features with the new image than other images in a previous category, or matching more strongly with the new image than other images in a previous category based on one or more values ​​indicating the one or more features. In some implementations, a clustering algorithm or the like is used to determine the category of the image. In some implementations, the network device 1016 obtains stock images from a predefined category having predefined features, which can be particularly useful in early implementations before sufficient user images are obtained.

[0137] In some implementations, the one or more values ​​represent features of the image. The wearable device 101 or the network device 1016 can identify and generate feature values ​​based on portions of an image, such as image 1004. An algorithm or trained model of the wearable device 101 or the network device 1016 can analyze pixel values ​​of an image, such as image 1004, and identify patterns. The patterns can indicate a particular type of object, object placement, or object appearance. In some implementations, feature identification includes object detection.

[0138] In some implementations, feature identification includes abstracting various elements of an image into categories of objects or visual qualities. For example, the network device 1016 may identify facial features of an image based on detecting a human face in the image. The network device 1016 may identify a category of the image based on one or more identified features, including facial features. If the network device 1016 identifies additional facial features indicative of a group of two or more people, the network device 1016 may identify the category of the image as a party scene. If the network device 1016 does not identify any other facial features and the faces are positioned to fill a threshold portion of the image, the network device 1016 may identify the category of the image as a portrait image.

[0139] In some implementations, the network device 1016 compares data corresponding to the image 1004 with data of one or more target images to identify which image of the one or more target images best matches the image 1004. For example, the network device 1016 can obtain one or more images of the same category as the image 1004. The one or more images of the same category can include one or more target images. The network device 1016 can generate a score each time it compares data corresponding to the image 1004 with data corresponding to one of the one or more images of the same category. The network device 1016 can determine a score, such as the highest or lowest depending on the implementation, that indicates the best match. The network device 1016 can identify the image of the one or more images of the same category that corresponds to the score that indicates the best match as the best matching image. In some implementations, the network device 1016 generates a distance measure that indicates a difference between one or more features of the target image and one or more features of the image 1004. For example, the score indicative of the best match may include one or more values ​​indicating that one or more features of the target image and one or more features of image 1004 are the same.

[0140] In some implementations, the cropping engine 1106 generates the cropped image 1018 based on the best matching image. For example, as discussed herein, the best matching image may be identified based on the network device 1016 comparing data corresponding to the image 1004 to data of one or more of the target images 1112. The network device 1016 may identify the cropping used in the best matching image and apply the cropping to the image 1004.

[0141] In some implementations, the feature engine 1110 can identify features indicative of cropping of the best matching image. For example, the feature engine 1110 can identify cropping features, such as a space between a human face and one or more edges of the image. Cropping features can include features identified in the target image 1112, such as the type, quality, or placement of a subject or object in the image.

[0142] In some implementations, the feature engine 1110 identifies a number of features for each image in the target images 1112. For example, the feature engine 1110 can identify a first number of features for a first image in the target images 1112 and a second number of features for a second image in the target images 1112. In some implementations, the feature engine 1110 can determine which features to include in the target image features 1116 based on comparing the first number of features to the second number of features. For example, the feature engine 1110 can identify features that are common to both the first number of features and the second number of features and include those features in the target image features 1116.

[0143] In some implementations, the feature engine 1110 generates weights for one or more features identified from the target images 1112. For example, the feature engine 1110 can identify a first feature in a first one of the target images 1112. The feature engine 1110 can generate a weight associated with the first feature that indicates the likelihood that the feature is indicative of a portion of the image that, when cropped, will result in a high quality image that is preferred by a user.

[0144] In some implementations, the feature engine 1110 can generate a weight for one or more features based on the number of similar features identified. For example, the feature engine 1110 can determine that a first identified feature is similar to a second identified feature from one or more images of the target images 1112. The feature engine 1110 can generate a weight for a feature of the target image features 1116 that corresponds to each of the first identified feature and the second identified feature, the weight indicating that the feature is more likely to be indicative of a portion of the image that, when cropped, will result in a high quality image preferred by the user compared to a feature identified only once from one or more images of the target images 1112.

[0145] In some implementations, the feature engine 1110 further increases the weight of a feature among the target image features if the feature is identified in at least two of the target images 1112. For example, the feature engine 1110 may generate a smaller weight for a feature identified twice in the same image compared to a feature identified once in two of the target images 1112. In general, the method of determining the features and weights of the target image features 1116 may vary based on the implementation.

[0146] The feature engine 1110 provides the target image features to the cropping engine 1106. In some implementations, the cropping engine 1106 obtains the target image features 1116 and provides the features 1116 to one or more algorithms or trained models, such as the model 1108. For example, the cropping engine 1106 can provide the target image features 1116 to the trained model 1108. In some implementations, the cropping engine 1106 provides both the image 1104 and the target image features 1116 to the trained model 1108. For example, the cropping engine 1106 can train the model 1108 to detect portions of an input image that include one or more features. The one or more features can be input parameters to the model 1108 or can be represented by weights and parameters of the model 1108. For example, the cropping engine 1106 can adjust one or more weights and parameters of the model 1108 based on one or more features of one or more input or target images 1112 .

[0147] In some implementations, the cropping engine 1106 trains the model 1108. For example, the cropping engine 1106 can provide an input image to the model 1108. The cropping engine 1106 can obtain output from the model 1108. The output can include predicted high quality portions of the input image. In some implementations, the cropping engine 1106 obtains ground truth data indicative of one or more high quality portions of the image. For example, a user can select portions of an image as high quality portions, such as portions that include features that the user likes, considers noteworthy, or interesting, among other things.

[0148] In another example, the ground truth data includes data representing one or more features of the target image features 1116. The cropping engine 1106 can determine whether a predicted high quality portion in an output of the model 1108 includes one or more features of the target image features 1116. The cropping engine 1106 can adjust one or more weights or parameters of the model 1108 based on comparing the features of the predicted high quality portion to the features of the target image features 1116.

[0149] In some implementations, the cropping engine 1106 identifies features from the predicted high quality portions of the image 1104. For example, the cropping engine 1106 can provide output from an algorithm or trained model, such as model 1108, to the feature engine 1110. The feature engine 1110 can identify which features are represented within the predicted high quality portions of the image 1104 based on processing of one or more algorithms or one or more trained models, such as model 1114.

[0150] In some implementations, the cropping engine 1106 identifies multiple portions of the image 1104 as potential high quality portions of the image. For example, the cropping engine 1106 can identify a first portion and a second portion of the image 1104. The cropping engine 1106 can identify one or more features represented in the first portion and one or more features represented in the second portion. The cropping engine 1106 can compare the one or more features represented in the first portion to target image features 1116. The cropping engine 1106 compares the one or more features represented in the second portion to target image features 1116. The cropping engine 1106 can determine that the first portion of the image 1104 is likely to be of higher quality than the second portion based on comparing the one or more features of the first portion and the second portion.

[0151] In some implementations, the cropping engine 1106 can determine the quality of a portion of an image based on features represented in the portion. For example, the cropping engine 1106 can obtain a first portion of the image 1104 and identify one or more features of the first portion through processing of one or more algorithms or trained models. The cropping engine 1106 can obtain target image features, such as target image features 1116, and compare the target image features to one or more features of the first portion. The cropping engine 1106 can determine whether the first portion is likely to be of high quality or low quality based on the comparison. In general, the high quality portion can include more features in common with the target image features 1116 than the low quality portion, indicating that the high quality portion is more similar to the target image 1112.

[0152] The cropping engine 1106 generates the image 1018. In some implementations, the cropping engine 1106 identifies multiple portions of the image 1104 and generates the image 1018 based on a portion of the multiple portions. For example, the cropping engine 1106 can compare features of the multiple portions to determine one or more portions of higher quality. The cropping engine 1106 can identify a single portion or multiple portions that may be of higher quality than one or more other identified portions. As discussed herein, the cropping engine 1106 can determine the quality based on identifying features represented within the portions.

[0153] In some implementations, the cropping engine 1106 identifies the portion of the image 1104 that is of the highest quality. For example, the cropping engine 1106 can determine a quality for each of the one or more identified portions of the image 1104. The cropping engine 1106 can compare the quality of each of the one or more identified portions. The cropping engine 1106 can select the identified portion that corresponds to the highest quality. The cropping engine 1106 can generate the image 1018 based on the identified portion that corresponds to the highest quality.

[0154] In some implementations, the network device 1016 provides image data to one or more trained models. In some implementations, the network device 1016 provides image data to one or more algorithms. For example, the network device 1016 can generate the image 1018 using one or more algorithms instead of or in addition to one or more of the models 1102, 1108, 1114. The one or more algorithms can include one or more algorithms for selecting a portion of the image 1004 or a portion of an image generated based on the image 1004, such as an upscaled version of the image 1004. The one or more algorithms can include one or more algorithms for generating an upscaled version of the image 1004. The one or more algorithms can include one or more algorithms for generating the target image features 1116. The one or more algorithms can include one or more algorithms for generating the image 1018.

[0155] In some implementations, the network device 1016 obtains data representing additional images to inform the indicated crop. For example, a user may select one or more images. The network device 1016 may obtain data indicating the one or more selected images. The network device 1016 may add the one or more selected images to the target image 1112. In some implementations, the feature engine 1110 generates new features of the target image features 1116 that represent the features of the added one or more selected images. In some implementations, the network device 1016 and components thereon retrain one or more models based on the network device 1016 adding the one or more selected images to the target image 1112. For example, the feature engine 1110 may retrain the model 1114 to generate features represented in the target image 1112 including the newly added images. In this manner, the system 1100 may update the type of cropped images generated to match the user's current preferences.

[0156] In some implementations, the network device 1016 retrieves one or more images from a network server, a user device, or an automated process. For example, the network device 1016 can retrieve images that were not selected by a user to be included in the target image 1112. Images that were not selected by a user and included in the target image 1112 can include images that are determined to be of high quality by a system administrator or an automated process. These high quality images can include, among other things, images that prominently feature human faces, images that are in focus, images that include subjects positioned in the frame with minimal background, aligned images, images that include visually appealing color and light balancing, and images taken by a professional photographer. In this manner, a system administrator or an automated process, such as one or more algorithms or trained models, can seed the target image 1112 to generate professional looking or high quality images.

[0157] In some implementations, the target image features 1116 include pre-configured features. For example, before or after deployment, a system administrator or an automated process can update the target image features 1116 to include one or more common features of high quality images, such as a subject in focus, a subject with eyes open, a subject looking at a camera that takes the image, such as the image capture device 1002, a particular subject, such as the user's pet, the user's friend, the user's family, a significant other, among others.

[0158] In some implementations, the network device 1016 obtains one or more data streams for determining an object or subject of interest. For example, the network device 1016 can analyze an image database representing images of a user of the image capture device 1002. The network device 1016 can identify one or more people in one or more images of the image database. The network device 1016 can include the particular identified person as a feature of the target image features 1116. For example, the target image features 1116 can include one or more features of the particular person. The cropping engine 1106 can then identify a portion of the image 1104 that includes one or more features representative of the one or more identified people. In this manner, the system 1100 can generate a cropped image that includes the person or object of interest.

[0159] Further Exemplary Processes 12 is a flow diagram of an exemplary process 1200 for generating a cropped image, according to one embodiment. The process 1200 can be implemented using the wearable multimedia device 101 described with reference to FIGS. 1-9.

[0160] The process 1200 includes obtaining 1202 an image captured using a camera coupled to a wearable multimedia device. For example, as shown in FIG. 11, an image capture device 1002 provides an image 1004 to a network device 1016. The image capture device 1002 may include a camera. The image capture device 1002 may be coupled to the wearable device 101.

[0161] The process 1200 includes comparing (1204) the image to one or more known images. For example, the network device 1016 or a component such as the cropping engine 1106 or the feature engine 1110 can compare the image to one or more of the target images 1112. The target images 1112 can include an image selected by a user. The target images 1112 can include one or more images from a user library or stock images available on a connected database. The target images 1112 can include images similar to the captured image 1004, such as images in the same category or images containing one or more similar or identical features.

[0162] In some implementations, the network device 1016 compares the image to the target image based on the extracted features. For example, instead of directly comparing the images, the network device 1016 can use the feature engine 1110 to generate features of the target image 1112 as well as the image 1004. The network device 1016 can compare the features of the target image 1112 to the features of the image 1004 to identify an image corresponding to a set of one or more features that best matches the image 1004. The network device 1016 can provide data corresponding to the best matching image, such as the best matching image or corresponding features of the target image 1112, to the cropping engine 1106. The cropping engine 1106 can then apply cropping to the captured image based on the data of the best matching image to generate a cropped image 1018. The image 1018 can include the same number or more common features as the best matching image. In general, the cropping of the cropping engine 1106 may modify one or more features of the image 1004 to match features of a matching image.

[0163] The process 1200 includes identifying (1206) one or more target features in the image based on comparing the image to one or more known images. For example, the cropping engine 1106 may obtain data corresponding to the image 1004, such as the upscaled image 1104, and identify features in the data. In some cases, the cropping engine 1106 provides the data to the feature engine 1110 to identify features that correspond to the image 1004. The cropping engine 1106 may then identify which features of the image 1004 correspond to target features 1116 identified from the feature engine 1110 processing the target image 1112.

[0164] The process 1200 includes generating 1208 a new image that includes a portion of the image that includes the one or more target features. For example, the cropping engine 1106 can generate the image 1018 based on the image 1004 and one or more target features identified in the portion of the image 1004, represented by the image 1018. The target features in the image 1018 can include, among other things, a human face that is in focus, a face facing a camera, such as the image capture device 1002, and the placement of a subject within a central portion of a frame of the image 1018.

[0165] The process 1200 includes providing 1210 the new image to the wearable multimedia device for presentation on a display associated with the wearable multimedia device. For example, as shown in FIG. 10B, the wearable device 101 can capture an image 1018 and provide data corresponding to the image 1018 to the projector subsystem 832. The projector subsystem 832 can project light onto the surface of the user's palm 1026 to present the image 1018. In some implementations, the projector subsystem 832 projects the light onto other surfaces, such as a wall, the ground, among others.

[0166] The features described herein can be implemented in digital electronic circuitry, or in computer hardware, computer firmware, computer software, or in combinations of them. The features can be implemented as a computer program product tangibly embodied in an information carrier, for example a machine-readable storage device, for execution by a programmable processor. The method steps can be carried out by the programmable processor executing a program of instructions for achieving functions of the implementations described herein by operating on input data and generating output.

[0167] The features described herein may advantageously be implemented as one or more computer programs executable on a programmable system including at least one programmable processor coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to send data and instructions to the data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform a particular operation or bring about a particular result. A computer program may be written in any form of programming language, including compiled or interpreted languages ​​(e.g., Objective C, Java), and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0168] Processors suitable for executing a program of instructions include, by way of example, both general purpose and special purpose microprocessors, and the sole processor or one of multiple processors or cores of any kind of computer. Generally, a processor receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer can communicate with mass storage devices for storing data files. These mass storage devices can include magnetic disks, such as internal hard disks and removable disks, magneto-optical disks, and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices, such as EPROMs, EEPROMs, and flash memory devices, magnetic disks, such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated in ASICs (Application Specific Integrated Circuits). To enable interaction with a user, the features may be implemented on a computer having a display device, such as a CRT (cathode ray tube), LED (light emitting diode), or LCD (liquid crystal display) display or monitor, for displaying information to the creator, and a keyboard and pointing device, such as a mouse or trackball, by which the creator can input into the computer.

[0169] One or more features or steps of the disclosed embodiments may be implemented using an application programming interface (API). An API may define one or more parameters passed between a calling application and other software code (e.g., an operating system, a library routine, a function) that provides a service, provides data, or performs an operation or calculation. An API may be implemented as one or more calls in program code that send or receive one or more parameters through a parameter list or other structure based on a calling convention defined in an API specification. A parameter may be a constant, a key, a data structure, an object, an object class, a variable, a data type, a pointer, an array, a list, or another call. API calls and parameters may be implemented in any programming language. A programming language may define the vocabulary and calling conventions used by a programmer to access functions that support the API. In some implementations, an API call may report to an application the capabilities, such as input capabilities, output capabilities, processing capabilities, power capabilities, communication capabilities, etc., of the device on which the application runs.

[0170] Several implementations have been described above. However, it will be understood that various modifications may be made. Elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. In yet another example, the logic flow depicted in the figures does not require the particular order or sequence depicted to achieve desirable results. In addition, other steps may be provided or steps may be eliminated from the described flow, and other components may be added to or removed from the described systems. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0171] 101 Wearable multimedia devices, wearable devices 102 Cloud Computing Platform 103 Network 104 Third Party Application ("App") Developers 105 Third Party Platforms 106 Database 200 Data Processing System 201 Recorder 202 Video Buffer 203 Audio Buffer 204 Photo Buffer 205 Import Server 206 Data Storage 207 Video Processor 208 Audio Processor 209 Photo Processor 210 Third Party Processors 211 Image processing / editing applications 212 Image processing / editing applications 213 Speech to Text Applications 300 Data Processing Pipeline 301 Import Server 302 Video Processor 304 Recorder 305 Audio Processor 306 Apps 307 Video Processing App 308 Data Merge Process 400 Data Processing Pipeline 401 Import Server 402 Third Party Processors 404 Recorder 405 Audio Processor 406 App 407 Third Party Applications 408 User Callback Queue 600 processes 700 Architecture 702 Processors, Devices 704 Storage Devices 706 Network Interfaces, Devices 708 Computer-readable media, devices 710 Communication Channel 712 Operating Systems 714 Network Communication Module 716 Data Processing Instructions 718 Interface Instructions 800 Architectures, Devices, and Systems 802 Memory Interface 804 Central Processing Unit, Processor 806 Peripheral Device Interface 810 Motion Sensor 812 Biometric Sensors 814 Depth Sensor 815 Location Processor 816 Electronic Magnetometer 817 Environmental Sensor 820 Camera Subsystem, Camera / Video Subsystem 822 Optical Sensor 824 Communication Subsystem 826 Audio Subsystem 828 Speaker 830 Microphone 832 Projector Subsystem, Projection Subsystem 840 I / O Subsystem 842 Touch Controller 844 Another input controller, other input controller 846 Touch Surface 848 Other Input / Control Devices 850 Memory 852 Operating Systems 854 Communication Order 858 Sensor Processing Instructions 860 Recorder Command 901 Controller 902 Battery 903 Power Management Chip (PMIC) 904 Solid-state laser 905 XY Scanner 906 Driver 907 Memory 908 Digital to Analog Converter (DAC) 909 Analog-to-Digital Converter (ADC) 1000 Systems 1002 Image capture device 1004 images 1006 Object 1008 People 1010 people 1012 people 1014 People 1016 Network Devices 1018 Cropped Images 1026 User's Palm 1028 Camera Interface 1100 System 1102 trained models 1104 upscaled version, upscaled image 1106 trimming engine 1108 trained models 1110 Features Engine 1112 Target Image 1114 trained models 1116 Target Image Features 1200 processes

Claims

1. 1. A method performed by a computing device, comprising: Obtaining a captured image using a camera coupled to a wearable multimedia device; comparing said image to one or more known images; identifying one or more target features in the image based on comparing the image to the one or more known images; generating a new image comprising a portion of the image that includes the one or more target features; providing the new image to the wearable multimedia device for presentation on a display associated with the wearable multimedia device; A method comprising:

2. identifying a matching image among the one or more known images that best matches the image captured using the camera based on comparing the image to the one or more known images; identifying one or more features in the matching image; identifying the one or more target features in the image based on identifying the one or more features in the matching image, where at least one of the one or more target features is a feature of the one or more features in the matching image; 2. The method of claim 1, comprising:

3. identifying a matching image among the one or more known images that best matches the image captured using the camera, The method of claim 2 , comprising comparing the one or more features of the matching image to the one or more target features in the image.

4. The method of claim 1 , wherein the one or more known images include a reference image specified by a user of the wearable multimedia device.

5. the one or more target features include one or more features in the reference image, the method comprising: analyzing the reference image based at least on receiving a designation of the reference image by the user of the wearable multimedia device; identifying one or more features in the reference image based on the analysis; For each feature of the one or more features, determining one or more parameters; comparing said one or more parameters to respective threshold values; selecting the feature as a target feature upon determining that the one or more parameters satisfy the respective thresholds; 5. The method of claim 4, further comprising:

6. The method of claim 5 , wherein the one or more parameters include a focus level.

7. The method of claim 1 , wherein the one or more target features include a representation of an object.

8. The method of claim 7 , wherein the object comprises one or more of a human, an animal, or a vehicle.

9. providing the new image for display, The method of claim 1 , comprising using a laser projector of the wearable multimedia device to project the new image onto a display surface that is physically separate from the wearable multimedia device.

10. The method of claim 9 , wherein the display surface comprises one of a surface of a user's hand, a user's palm, a user's wrist, a table top, a wall, a portion of glass, or a portion of ground.

11. The method of claim 1 , wherein the computing device is a server communicatively coupled to the wearable multimedia device.

12. The method of claim 1 , wherein the computing device is the wearable multimedia device.

13. The method of claim 1 , comprising generating a higher resolution version of the image captured using the camera coupled to the wearable multimedia device.

14. generating a higher resolution version of the image, providing the image to a machine learning model; generating a higher resolution version of the image based on the obtained output from the machine learning model; 14. The method of claim 13, comprising:

15. One or more computers; one or more computer memory devices interoperably coupled to the one or more computers and having a tangible, non-transitory, machine-readable medium storing one or more instructions, which when executed by the one or more computers: Obtaining an image captured using a camera coupled to a wearable multimedia device; comparing the image to one or more known images; identifying one or more target features in the image based on comparing the image to the one or more known images; generating a new image that includes a portion of the image that includes the one or more target features; providing the new image to the wearable multimedia device for presentation on a display associated with the wearable multimedia device; one or more computer memory devices for performing one or more operations including A computer-implemented system comprising:

16. The operation, identifying a matching image among the one or more known images that best matches the image captured using the camera based on comparing the image to the one or more known images; identifying one or more features in the matching image; identifying the one or more target features in the image based on identifying the one or more features in the matching image, wherein at least one of the one or more target features is a feature of the one or more features in the matching image; The system of claim 15, comprising:

17. identifying a matching image among the one or more known images that best matches the image captured using the camera; The system of claim 16 , further comprising comparing the one or more features of the matching image to the one or more target features in the image.

18. The system of claim 15 , wherein the one or more known images include a reference image specified by a user of the wearable multimedia device.

19. the one or more target features include one or more features in the reference image, and the operation comprises: analyzing the reference image based at least on receiving a designation of the reference image by the user of the wearable multimedia device; identifying one or more features in the reference image based on the analysis; and For each feature of the one or more features, Determining one or more parameters; comparing said one or more parameters to respective threshold values; selecting the feature as a target feature upon determining that the one or more parameters satisfy the respective thresholds; 20. The system of claim 18, further comprising:

20. One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to: Obtaining an image captured using a camera coupled to a wearable multimedia device; comparing the image to one or more known images; identifying one or more target features in the image based on comparing the image to the one or more known images; generating a new image that includes a portion of the image that includes the one or more target features; providing the new image to the wearable multimedia device for presentation on a display associated with the wearable multimedia device; One or more non-transitory computer-readable media for causing operations to be performed, including:

Images