Machine learning-based image compression settings that reflect user preferences

Abstract

[0003] Embodiments described herein relate to a method, a device, and a computer-readable medium for determining compression settings. An input image may be obtained when associated with a user account. One or more features of the input image may be determined using a feature detection machine learning model. Compression settings for the input image may be determined based on one or more features in the input image using a user-specific machine learning model personalized to the user account. The input image may be compressed based on the compression settings.

Description

[Background technology]

[0001] background As smartphones and other portable cameras become more popular, users are capturing more and more images. However, on-device, as well as cloud or server storage, are finite resources. Image compression is an effective way to reduce the amount of storage space required to store images. However, lossy compression can result in compressed images with perceptibly lower quality, resulting in a less than optimal user experience.

[0002] The discussion of the background art provided herein is intended to generally present the context of the present disclosure. The work of the presently named inventors, to the extent that it is described in this background art section, is not admitted expressly or impliedly as prior art to the present disclosure, as are aspects of the description that may not be considered prior art at the time of filing. Summary of the Invention [Means for solving the problem]

[0003] overview

[0003] Embodiments described herein relate to methods, devices, and computer-readable media for generating compression settings. The method may include obtaining an input image associated with a user account, determining one or more features of the input image using a feature detection machine learning model, determining a compression setting for the input image based on the one or more features in the input image using a user-specific machine learning model personalized to the user account, and compressing the input image based on the compression setting.

[0004] In some embodiments, the feature detection machine learning model is generated by obtaining a training set of digital images and corresponding features, and training the feature detection machine learning model based on the training set and the corresponding features, where after training, the feature detection machine learning model is capable of identifying image features in an input image provided to the feature detection machine learning model. In some embodiments, the feature detection machine learning model includes a convolutional neural network (CNN) having multiple network layers, where each network layer extracts one or more image features at a different level of abstraction. In some embodiments, the user-specific machine learning model is generated by obtaining a training set of user-specific features associated with a user indicative of a user action with reference to one or more previous images, and training the user-specific machine learning model based on the user-specific features and the one or more previous images, where after training, the user-specific machine learning model determines a rating of an input image provided to the user-specific machine learning model. In some embodiments, the image features of each of the one or more previous images are obtained by applying the feature detection machine learning model to the one or more previous images.

[0005] In some embodiments, the method may further comprise providing a first user interface to a user associated with a user account having two or more versions of a sample image each compressed with a different compression setting, obtaining user input from the user identifying a particular version of the sample image, and selecting a compression setting associated with the particular version of the sample image as a baseline compression setting for the user account. In some embodiments, determining the compression setting includes the user-specific machine learning model determining a rating of the input image and mapping the rating to a compression setting, the mapping based on the baseline compression setting.

[0006] In some embodiments, the method may further comprise determining that a rating of the input image meets an importance threshold, and, in response to determining that the rating meets the importance threshold, performing one or more of: providing a suggestion for the user to share the input image, prioritizing a backup of the input image over backups of other images associated with the user account that do not meet the importance threshold, or providing a second user interface including instructions for capture of a subsequent image if a scene depicted in the subsequent image has at least one of the one or more characteristics of the input image.

[0007] Some embodiments may include a computing device comprising a processor and a memory storing instructions that, when executed by the processor, cause the processor to perform operations including obtaining an input image associated with a user account; determining one or more features of the input image using a feature detection machine learning model; determining compression settings for the input image based on the one or more features in the input image using a user-specific machine learning model personalized to the user account; and compressing the input image based on the compression settings.

[0008] In some embodiments, the feature detection machine learning model is generated by obtaining a training set of digital images and corresponding features and training the feature detection machine learning model based on the training set and the corresponding features, where after training, the feature detection machine learning model is capable of identifying image features in an input image provided to the feature detection machine learning model. In some embodiments, the user-specific machine learning model is generated by obtaining a training set of user-specific features associated with a user indicative of a user action with reference to one or more prior images and training the user-specific machine learning model based on the user-specific features and the one or more prior images, where after training, the user-specific machine learning model determines a rating for the input image provided to the user-specific machine learning model.

[0009] In some embodiments, the memory further stores instructions that, when executed by the processor, cause the processor to perform yet another operation including providing a first user interface to a user associated with a user account having two or more versions of a sample image, each compressed with a different compression setting, obtaining user input from the user identifying a particular version of the sample image, and selecting a compression setting associated with the particular version of the sample image as a baseline compression setting for the user account. In some embodiments, determining the compression setting includes the user-specific machine learning model determining a rating of the input image and mapping the rating to a compression setting, the mapping based on the baseline compression setting.

[0010] Some embodiments may comprise a non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more computers, cause the one or more computers to perform operations including obtaining an input image associated with a user account; determining one or more features of the input image using a feature detection machine learning model; determining compression settings for the input image based on the one or more features in the input image using a user-specific machine learning model personalized to the user account; and compressing the input image based on the compression settings.

[0011] In some embodiments, the feature detection machine learning model is generated by obtaining a training set of digital images and corresponding features and training the feature detection machine learning model based on the training set and the corresponding features, where after training, the feature detection machine learning model is capable of identifying image features in an input image provided to the feature detection machine learning model. In some embodiments, the user-specific machine learning model is generated by obtaining a training set of user-specific features associated with a user indicative of a user action with reference to one or more prior images and training the user-specific machine learning model based on the user-specific features and the one or more prior images, where after training, the user-specific machine learning model determines a rating for an input image provided to the user-specific machine learning model. In some embodiments, the training set further includes image features for each of the one or more prior images.

[0012] In some embodiments, the operations further include providing a first user interface to a user associated with a user account having two or more versions of a sample image each compressed with a different compression setting, obtaining user input from the user identifying a particular version of the sample image, and selecting a compression setting associated with the particular version of the sample image as a baseline compression setting for the user account. In some embodiments, determining the compression setting includes the user-specific machine learning model determining a rating of the input image and mapping the rating to a compression setting, the mapping based on the baseline compression setting. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram illustrating an example of a network environment that may be used in one or more embodiments described herein. [Diagram 2] FIG. 1 is a block diagram illustrating an example of a device that may be used in one or more embodiments described herein. [Diagram 3]FIG. 1 is a flow diagram illustrating an example of a method using a feature detection machine learning model to identify one or more features in an input image and using a user-specific machine learning model to determine compression settings from the one or more features of the image, according to some embodiments. [Figure 4] FIG. 1 is a flow diagram illustrating an example of a method for creating a training model according to some embodiments. [Diagram 5] FIG. 2 is a flow diagram illustrating an example of a method for applying a model to an input image according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Detailed Description A user captures images using a camera, such as via a smartphone or other device. For example, the images may include still images, cinemagraphs / movie images, or image frames from a video. The user may store the images on a client device or a server, e.g., a server that provides an image hosting service. Applications may be provided via the user's client device and / or server that allow the user to manage the images, such as viewing and / or editing the images, generating image-based creations such as images in slideshows, collages, and the like, sharing the images, posting the images to social network or chat applications, and allowing other users to provide indications of approval for the images, such as by "liking" or placing on the image.

[0015] Storage space on a client device or on a server is limited. One way to obtain additional storage space without deleting images is to use image compression to reduce the file size of the images. However, lossy compression degrades image quality and may result in a less than optimal user experience when all images associated with a user are subjected to image compression.

[0016] If a user account has a large number of images, a subset of those images are likely to be images that the user particularly likes. For example, some users may be picky about the quality of their landscape photos because they enjoy printing them. Others may be picky about high-resolution portraits of people because they enjoy sharing these portraits with their families or may run a photography business. The image quality of receipts, screenshots, or other functional images (e.g., meme images, business card photos, newspaper articles, etc.) may not be as important. Thus, a user's perception of loss due to image compression may depend on the type of image and the user account associated with the image. Therefore, it is advantageous to identify which images are likely to be important to the user. In some embodiments, the image management application generates and utilizes a feature detection machine learning model that identifies features in the first image. The image management application may also generate a user-specific machine learning model personalized to the user account. The image management application may use the user-specific machine learning model to determine compression settings for the first image. The image management application may compress the first image based on a compression setting to free up storage space because the resulting compressed image is a smaller file size than the original image.

[0017] The drawings use like reference numbers to identify like elements. Letters following a reference number, such as "103a," indicate that the text is specifically referring to the element having that particular reference number. A reference number in the text without letters following it, such as "103," refers to any or all of the elements in the figures having that reference number (e.g., "103" in the text refers to reference numbers "103a" and / or "103b" in the figures).

[0018] 100 examples of network environments 1 is a block diagram illustrating an example of a network environment 100 that may be used in some embodiments described herein. In some embodiments, the network environment 100 includes one or more server systems, such as server system 101 in the example of FIG. 1. Server system 101 may be in communication with a network 105, for example. Server system 101 may include a server device 104 and a database 199 or other storage device. Database 199 may store one or more images and / or videos and metadata associated with the one or more images and / or videos. In some embodiments, server device 104 may provide an image management application 103a. Image management application 103a may access images stored in database 199.

[0019] The network environment 100 may also include one or more client devices, e.g., client devices 115a, 115n, that may communicate with each other and / or with the server system 101 via the network 105. The network 105 may be any type of communication network, including one or more of the Internet, a local area network (LAN), a wireless network, a switched or hub connection, and the like. In some embodiments, the network 105 may include peer-to-peer communication between devices, for example, using a peer-to-peer wireless protocol (e.g., Bluetooth, Wi-Fi Direct, Ultrawideband, and the like). An example of peer-to-peer communication between two client devices 115a and 115b is illustrated by arrow 132.

[0020] For ease of explanation, FIG. 1 shows one block for server system 101, server device 104, and database 199, and two blocks for client devices 115a and 115n. Server blocks 101, 104, and 199 may represent multiple systems, server devices, and network databases, and the blocks may be provided in configurations different than those shown. For example, server system 101 may represent multiple server systems that can communicate with other server systems over network 105. In some embodiments, server system 101 may include, for example, a cloud hosting server. In some examples, database 199 and / or other storage devices may be provided in a server system block(s) separate from server device 104, and may communicate with server device 104 and other server systems over network 105.

[0021] There may be any number of client devices 115. Each client device may be any type of electronic device, for example, a desktop computer, a laptop computer, a portable or mobile device, a mobile phone, a smartphone, a tablet computer, a camera, a smart display, a television, a TV set-top box or entertainment device, a wearable device (e.g., display glasses or goggles, a watch, a headset, an armband, jewelry, etc.), a personal digital assistant (PDA), a media player, a gaming device, etc. Some client devices may also include a local database similar to database 199 or other storage. In some embodiments, network environment 100 may not have all of the components shown and / or may have other elements instead of or in addition to those described herein, including other types of elements.

[0022] In various embodiments, users 125 may communicate with server system 101 and / or with each other using their respective client devices 115a, 115n. In some examples, users 125 may interact with each other through applications running on their respective client devices and / or server system 101, through network services implemented on server system 101, e.g., social networking services, image hosting services, or other types of network services. For example, each client device 115a, 115n may communicate data to or from one or more server systems, e.g., server system 101.

[0023] In some embodiments, the server system 101 may provide appropriate data to the client devices 115a, 115n so that each client device 115 can receive communication or shared content uploaded to the server system 101 and / or the network service. In some examples, the users 125 can interact via audio or video conferencing, audio, video, or text chat, or other communication modes or applications. The network service implemented by the server system 101 may include systems that enable the users 125 to perform various communications, form links and associations, upload and post shared content such as images, text, video, audio, and other types of content, and / or perform other functions. For example, the client device 115 can display received data, such as content posts that are transmitted or streamed to the client device 115 and originate from different client devices 115 (or directly from different client devices 115) via the server and / or network service, or originate from the server system 101 and / or the network service. In some embodiments, client devices 115a, 115n may communicate directly with one another, for example, using peer-to-peer communication between client devices 115, 115n as described above. In some embodiments, a "user" may include a person interfacing with system or network 105 in addition to one or more programs or virtual entities.

[0024] In some embodiments, either of the client devices 115a, 115n may provide one or more applications. For example, as shown in FIG. 1, the client device 115a may provide a camera application 152 and an image management application 103b. The client device 115n may also provide similar applications. The camera application 152 may provide the user 125a of the respective client device 115a with the ability to capture images using the camera of the respective client device 115a. For example, the camera application 152 may be a software application that executes on the client device 115a.

[0025] In some embodiments, the camera application 152 may provide a user interface. For example, the user interface may enable a user of the client device 115a to select an image capture mode, e.g., a still image (or photo) mode, a burst mode (e.g., capturing a number of consecutive images in a short period of time), a motion picture mode, a video mode, a high dynamic range (HDR) mode, a resolution setting, etc. For example, the video mode may correspond to the capture of a video that includes multiple frames and may be of any length. Additionally, the video mode may support different frame rates, e.g., 25 frames per second (fps), 30 fps, 50 fps, 60 fps, etc. One or more parameters of the image capture may be changed during the capture of an image or video. For example, a user may use the client device 115a to zoom in or out on a scene while capturing a video.

[0026] In some embodiments, camera application 152 may implement (e.g., partially or entirely) the methods described herein with reference to Figures 3 and 4A-4B. In some embodiments, image management application 103a and / or image management application 103b may implement (e.g., partially or entirely) the methods described herein with reference to Figures 3 and 4A-4B.

[0027] The camera application 152 and the image management application 103b may be implemented using hardware and / or software on the client device 115a. In different embodiments, the image management application 103b may be a standalone application running on either the client device 115a, 115n, for example, or may operate in conjunction with an image management application 103a provided on the server system 101.

[0028] With the user's permission, the image management application 103 may perform one or more automatic functions, such as storing the image or video (e.g., backing up to database 199 of server system 101), enhancing the image or video, stabilizing the image or video, recognizing one or more features in the image, e.g., face, body, type of object, type of motion, compressing the image, etc. In some examples, image or video stabilization may be performed based on input from an accelerometer, gyroscope, or other sensor of the client device 115a and / or based on a comparison of multiple frames of a movie or video.

[0029] The image management application 103 may also provide image management functionality such as displaying images and / or videos in a user interface (e.g., in a one-up view with a single image, in a grid view with multiple images), editing an image or video (e.g., adjusting image settings, applying filters, changing image focus, removing one or more frames of a movie or video), sharing images with other users (e.g., of client devices 115a, 115n), archiving images (e.g., saving images that do not appear in the primary user interface), creating image-based creations (e.g., animated motion-based artifacts such as collages, photo books, animations, stories, video loops, etc.), etc. In some embodiments, to create image-based creations, the image management application 103 may utilize one or more labels associated with an image or video.

[0030] In some embodiments, image management application 103 may determine one or more characteristics of the image and determine compression settings for the image based on the one or more characteristics in the image. In some embodiments, image management application 103 may store the compression settings associated with the image or video and the compressed image or video in database 199 and / or a local database (not shown) on client device 115. In some embodiments, image management application 103 deletes the original image immediately, saves the original image and prompts the user to confirm the deletion, or saves the original image for a certain number of days and then deletes the original image.

[0031] User interfaces on client devices 115 may enable the display of user content and other content, including images, videos, data, and other content, as well as communications, privacy settings, notifications, and other data. Such user interfaces may be displayed using software on client devices 115, software on server device 104, and / or a combination of client and server software running on server device 104, e.g., application software or client software in communication with server system 101. User interfaces may be displayed by a display device of client device 115 or server device 104, e.g., a touch screen or other display screen, projector, etc. In some embodiments, application programs running on server system 101 may communicate with client devices 115 to receive user input at client devices 115 and output data, such as visual data, audio data, etc., at client devices 115.

[0032] In some embodiments, the server system 101 and / or any of the one or more client devices 115a, 115n may provide a communications application program. The communications program may enable a system (e.g., the client device 115 or the server system 101) to provide options for communicating with other devices. The communications program may provide one or more associated user interfaces that are displayed on a display device associated with the server system 101 or the client device 115. The user interfaces may provide the user with various options for selecting a communication mode, a user or device to communicate with, etc. In some examples, the communications program may provide an option to transmit or broadcast the content post, for example to a broadcast area, and / or output a notification indicating that the content post has been received by the device and, for example, that the device is within a broadcast area defined for the post. The communications program may display or otherwise output the transmitted and received content posts, for example, in any of a variety of formats. The content posts may include, for example, images to be shared with other users.

[0033] Other embodiments of the features described herein may use any type of system and / or service. For example, other network services (e.g., connected to the Internet) may be used instead of or in addition to a social networking service. Any type of electronic device may utilize the features described herein. Some embodiments may provide one or more features described herein on one or more client or server devices that are disconnected from or intermittently connected to a computer network. In some examples, a client device 115 that includes or is connected to a display device may display data (e.g., content) that is stored in a storage device local to the client device 115 and that was previously received, e.g., over a communication network.

[0034] Device 200 Example 2 is a block diagram illustrating an example of a device 200 that may be used to implement one or more features described herein. In one example, the device 200 may be used to implement a client device 115, such as any of the client devices 115a, 115n shown in FIG. 1. Alternatively, the device 200 may implement a server device, such as the server device 104 shown in FIG. 1. In some embodiments, the device 200 may be used to implement a client device, a server device, or both a client device and a server device. The device 200 may be any suitable computer system, server, or other electronic or hardware device as described above.

[0035] One or more methods described herein may be implemented as a standalone program that may run on any type of computing device, a program running on a web browser, a mobile application ("app") running on a mobile computing device (e.g., a mobile phone, a smartphone, a smart display, a tablet computer, a wearable device (watch, armband, jewelry, headwear, virtual reality goggles or glasses, augmented reality goggles or glasses, head mounted displays, etc.), a laptop computer, etc.). In one example, a client / server architecture may be used, e.g., a mobile computing device (as a client device) sends user input data to a server device and receives final output data for output (e.g., for display) from the server. In another example, all computations may be performed within the mobile application (and / or other applications) on the mobile computing device. In another example, computations may be split between the mobile computing device and one or more server devices.

[0036] In some embodiments, device 200 includes a processor 202, a memory 204, an input / output (I / O) interface 206, a camera 208, and a display device 210. Processor 202 may be one or more processors and / or processing circuits for executing program code and controlling basic operations of device 200. A "processor" includes any suitable hardware system, mechanism, or component that processes data, signals, or other information. A processor may include a general-purpose central processing unit (CPU) having one or more cores (e.g., single-core, dual-core, or multi-core configurations), multiple processing units (e.g., multi-processor configurations), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit for implementing functions, a dedicated processor implementing neural network model-based processing, a neural circuit, a processor optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some embodiments, processor 202 may include one or more co-processors implementing neural network processing. In some embodiments, the processor 202 may be a processor that processes data to generate a probabilistic output, e.g., the output generated by the processor 202 may be inaccurate or accurate within a range from an expected output. The processing need not be limited to a particular geographic location, nor is it time-limited. For example, the processor may perform its functions in real-time, offline, in batch mode, etc. Parts of the processing may be performed by different (or the same) processing systems at different times, in different locations, etc. A computer may be any processor in communication with a memory.

[0037] Memory 204 is typically provided within device 200 for access by processor 202, may be any suitable processor-readable storage medium, such as random access memory (RAM), read only memory (ROM), electrically erasable read only memory (EEPROM), flash memory, etc., suitable for storing instructions for execution by a processor or set of processors, and may be located separately from and / or integrated with processor 202. Memory 204 may store software run on server device 200 by processor 202, including operating system 212, other applications 214, application data 216, and image management application 103.

[0038] The other applications 214 may include applications such as a camera application, an image gallery or image library application, an image management application, a data display engine, a web hosting engine or application, an image display engine or application, a media display application, a communication engine, a notification engine, a social networking engine, a media sharing application, a mapping application, etc. One or more methods disclosed herein are operable in several environments and platforms, for example, as a standalone computer program that can be executed on any type of computing device, as a web application having a web page, as a mobile application ("app") that runs on a mobile computing device, etc. In some embodiments, the other applications 214 may each include instructions that enable the processor 202 to perform some or all of the functions described herein, e.g., the methods of FIGS. 3 and 4A-4B.

[0039] Application data 216 may be data generated by other applications 214 or hardware for device 200. For example, application data 216 may include images captured by camera 208, user actions identified by other applications 214 (e.g., social networking applications), etc.

[0040] The input / output interface 206 can provide functionality that allows the device 200 to interface with other systems and devices. The interfaced devices can be included as part of the device 200 or can be separate and in communication with the device 200. For example, network communication devices, storage devices (e.g., memory and / or database 199), and input / output devices can communicate through the input / output interface 206. In some embodiments, the input / output interface can be connected to interface devices such as input devices (keyboards, pointing devices, touch screens, microphones, cameras, scanners, sensors, etc.) and / or output devices (display devices, speaker devices, printers, motors, etc.).

[0041] Some examples of interfaced devices that may be connected to the input / output interface 206 may include one or more display devices 210 that may be used to display content, e.g., images, videos, and / or user interfaces of output applications as described herein. The display devices 210 may be connected to the device 200 via a local connection (e.g., a display bus) and / or via a network connection and may be any suitable display device. The display devices 210 may include any suitable display device, such as a liquid crystal display (LCD), light emitting diode (LED), or plasma display screen, a cathode ray tube (CRT), a television, a monitor, a touch screen, a three-dimensional display screen, or other visual display device. For example, the display devices 210 may be a flat display screen provided on a mobile device, multiple display screens embedded in a glasses form factor or headset device, or a monitor screen for a computing device.

[0042] The input / output interface 206 can interface to other input and output devices. Some examples include one or more cameras, such as camera 208, that can capture images. Some embodiments can provide a microphone for capturing sound (e.g., as part of a captured image, as a voice command, etc.), an audio speaker device for outputting sound, or other input and output devices.

[0043] The camera 208 may be any type of camera capable of capturing video including multiple frames. As used herein, a camera may include any image capture device. In some embodiments, the camera 208 may include multiple lenses with different capabilities, e.g., forward facing vs. rear facing, different zoom levels, image resolution of the captured images, etc. In some embodiments, the device 200 may include one or more sensors, such as a depth sensor, an accelerometer, a position sensor (e.g., a global positioning system (GPS)), a gyroscope, etc. In some embodiments, one or more sensors may operate in conjunction with the camera 208 to obtain sensor readings corresponding to different frames of video captured using the camera 208.

[0044] 103 Examples of Image Management Applications The image management application 103 may include a feature detection machine learning module 218 , a user-specific machine learning module 220 , a compression module 222 , and a user interface module 224 .

[0045] In some embodiments, the feature detection machine learning module 218 generates a feature detection machine learning model to identify features from images. For example, features may be vectors in a multidimensional feature space (embedding). Images with similar features may have similar feature vectors, e.g., the vector distance between the feature vectors of such images may be less than the vector distance between different images. The feature space may be a function of various factors of the image, such as the subject matter depicted (objects detected in the image), image composition, color information, image orientation, image metadata, specific objects recognized in the image (e.g., known faces with user permission), etc.

[0046] The user-specific machine learning module 220 may generate a user-specific machine learning model personalized for a user account associated with the user. The user-specific machine learning module 220 may use the user-specific machine learning model to determine compression settings for an image based on features in the image. This advantageously conserves storage space while preserving the highest possible quality of the image based on what the user is interested in. For example, the user-specific machine learning model may output an indication that an image of a sunset should be compressed with a high compression ratio. For example, such an indication may be determined based on the user-specific machine learning model analyzing the image, including features detected by the feature detection machine learning module 218. Because the image includes a sunset, if the image is not associated with features that the user determines are not important to the image, the compression setting should be the highest level of compression for the image.

[0047] Feature detection machine learning module 218 example The feature detection machine learning module 218 generates a feature detection machine learning model that determines one or more features of the input image. In some embodiments, the feature detection machine learning module 218 includes a set of instructions executable by the processor 202 to generate the feature detection machine learning model. In some embodiments, the feature detection machine learning module 218 may be stored in the memory 204 of the device 200 and accessible and executable by the processor 202.

[0048] In some embodiments, the feature detection machine learning module 218 may use training data to generate a trained model, specifically, a feature detection machine learning model. For example, the training data may include any type of data, such as images (e.g., still images, cinemagraphs / movies, or image frames from a video, etc.) and, optionally, corresponding features (e.g., labels or tags associated with each of the images that identify objects in the images).

[0049] For example, the training data may include a training set including multiple digital images and corresponding features. In some embodiments, the training data may include images with augmentations such as rotation, light shift, and color shift to provide invariance to the model when user photos are provided that may be rotated or have unusual characteristics (e.g., artifacts of the camera used to capture the image). The training data may be obtained from any source, e.g., a data repository specifically marked for training, data for which permission is provided for use as training data for machine learning, etc. In embodiments in which one or more users authorize the use of their respective user data to train the machine learning model, the training data may include such user data. In embodiments in which users authorize the use of their respective user data, the data may include authorized data such as images / videos or image / video metadata (e.g., images, corresponding features that may result from users providing manual tags or labels, etc.), communications (e.g., messages on social networks; emails; chat data such as text messages, audio, video, etc.), documents (e.g., spreadsheets, text documents, presentations, etc.), etc.

[0050] In some embodiments, the training data may include data that is not based on user input or activity in the context being trained, e.g., synthetic data generated for training purposes, such as data generated from a simulation or computer-generated image / video. In some embodiments, the feature detection machine learning module 218 uses weights imported and unedited / transferred from another application. For example, in these embodiments, the trained model may be generated, for example, on a different device and provided as part of the image management application 103. In various embodiments, the trained model may be provided as a data file that includes a model structure or format (e.g., defining the number and type of neural network nodes, the connectivity between the nodes, and the organization of the nodes into multiple layers) and associated weights. The feature detection machine learning module 218 may read the data file for the trained model and implement a neural network with node connectivity, layers, and weights based on the model structure or format specified in the trained model.

[0051] The feature detection machine learning module 218 generates a trained model, referred to herein as a feature detection machine learning model. In some embodiments, the feature detection machine learning module 218 is configured to apply the feature detection machine learning model to data, such as the application data 216 (e.g., an input image) to identify one or more features in the input image and generate a feature vector (embedding) that represents the image. In some embodiments, the feature detection machine learning module 218 may include software code executed by the processor 202. In some embodiments, the feature detection machine learning module 218 may specify a circuit configuration (e.g., in the case of a programmable processor, in the case of a field programmable gate array (FPGA), etc.) that enables the processor 202 to apply the feature detection machine learning model. In some embodiments, the feature detection machine learning module 218 may include software instructions, hardware instructions, or a combination. In some embodiments, the feature detection machine learning module 218 may provide an application programming interface (API) usable by the operating system 212 and / or other applications 214 to invoke the feature detection machine learning module 218, e.g., to apply the feature detection machine learning model to the application data 216 to determine one or more features of the input image.

[0052] In some embodiments, the feature detection machine learning model may include one or more model formats or structures. In some embodiments, the feature detection machine learning model may use a support vector machine, but in some embodiments, a convolutional neural network (CNN) is preferred. For example, the model format or structure may include any type of neural network, such as a linear network, a deep neural network implementing multiple layers (e.g., a "hidden layer" between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (CNN) (e.g., a network that splits or partitions input data into multiple portions or tiles, processes each tile separately using one or more neural network layers, and aggregates the results from the processing of each tile), a sequence-to-sequence neural network (e.g., a network that receives sequential data as input, such as words in a sentence, frames in a video, etc., and produces a sequence of results as output), etc.

[0053] The model format or structure may specify the connectivity between various nodes and the organization of the nodes into layers. For example, the nodes in the first layer (e.g., input layer) may receive data as input data or application data 216. Such data may include, for example, one or more pixels per node, for example, if the feature detection machine learning model is used for analysis of an input image, such as, for example, a first image associated with a user account. Subsequent intermediate layers may receive as input the output of the nodes of the preceding layer according to the connectivity specified in the model format or structure. These layers are sometimes referred to as hidden layers. The final layer (e.g., output layer) generates the output of the machine learning application. For example, the output may be image features associated with the input image. In some embodiments, the model format or structure also specifies the number and / or type of nodes in each layer.

[0054] The features output by the feature detection machine learning module 218 may include subject matter (e.g., a sunset vs. a particular person); colors present in the image (green hills vs. blue lake); color balance; light source, angle, and intensity; location of objects in the image (e.g., conforming to the rule of thirds); location of objects relative to each other (e.g., depth of field), shot position; focus (foreground vs. background); or shadows. While the features described above are human understandable, it will be understood that the features output may be embeddings or other mathematical values ​​representative of the image and are not human analyzable (e.g., individual feature values ​​may not correspond to specific features such as colors present, object positions, etc.). However, the trained model is robust to images such that similar features are output for similar images, and images with significant differences will have correspondingly different features.

[0055] In some embodiments, the model format is a CNN with network layers, each network layer extracts image features at a different level of abstraction. The CNN used to identify features in the image may be used for image classification. In some embodiments, the CNN may be used to identify features of the image, and then transfer learning is applied by replacing the classification layer, or more specifically, the fully connected feed-forward neural network output layer, with a user-specific machine learning model, as described below. In some embodiments, the CNN is a VGGnet, ResNet, AlexNet, Inception network, or any other state-of-the-art neural network considered for image processing applications, trained using a training set of digital images such as ImageNet. The model architecture may include combinations and orderings of layers consisting of multidimensional convolutions, average pooling, max pooling, activation functions, normalization, regularization, and other layers and modules actually used for the deep neural network applied.

[0056] In different embodiments, the feature detection machine learning model may include one or more models. One or more of the models may include a plurality of nodes or, in the case of a CNN, a filter bank arranged in layers per model structure or form. In some embodiments, the node may be, for example, a memoryless computational node configured to process a unit of input and generate a unit of output. The computation performed by the node may include, for example, multiplying each of a plurality of node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or intercept value to generate the node output.

[0057] In some embodiments, the computations performed by the nodes may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a non-linear function. In various embodiments, such computations may include operations such as matrix multiplication. In some embodiments, the computations by multiple nodes may be performed in parallel, e.g., using multiple processor cores of a multi-core processor, using individual processing units of a GPU, or using dedicated neural circuitry. In some embodiments, the nodes may include memory, e.g., to store and use one or more previous inputs in processing subsequent inputs. For example, a node with memory may include a long short-term memory (LSTM) node. An LSTM node may use memory to maintain state that allows the node to act like a finite state machine (FSM). Models with such nodes may be useful in processing continuous data, e.g., words in a sentence or paragraph, a series of images, frames in a video, speech or other audio, etc. For example, a heuristics-based model used in a gating model may store one or more previously generated features corresponding to a previous image.

[0058] In some embodiments, the feature detection machine learning model may include embeddings or weights for individual nodes. For example, the feature detection machine learning model may begin as a number of nodes organized into layers as specified by the model format or structure. At initialization, a respective weight may be applied to each pair of nodes that are connected according to the model format, e.g., the connections between nodes in successive layers of a neural network. For example, the respective weights may be randomly assigned or initialized to default values. The feature detection machine learning model may then be trained, e.g., using a training set of digital images, to generate results. In some embodiments, a subset of the entire architecture may be reused from other machine learning applications as a transfer learning approach to leverage the pre-trained weights.

[0059] For example, training may include application of supervised learning techniques. In supervised learning, training data may include multiple inputs (e.g., a set of digital images) and corresponding expected outputs for each input (e.g., one or more features for each image). Based on a comparison of the output of the feature detection machine learning model to the expected outputs, the values ​​of the weights are automatically adjusted, for example, to increase the probability that the feature detection machine learning model will generate the expected outputs when provided with similar inputs.

[0060] In some embodiments, training may include the application of unsupervised learning techniques, in which only input data (e.g., images with labeled features) may be provided, and a feature detection machine learning model may be trained to distinguish the data, for example, to cluster image features into multiple groups, each group containing images with features that are similar in some manner.

[0061] In various embodiments, the trained model includes a set of weights corresponding to the model structure. In embodiments where a training set of digital images is omitted, the feature detection machine learning module 218 may generate a feature detection machine learning model based on pre-training, e.g., by a developer of the feature detection machine learning module 218, a third party, etc. In some embodiments, the feature detection machine learning model may include a set of weights that is fixed, e.g., downloaded from a server that provides the weights.

[0062] In some embodiments, the feature detection machine learning module 218 may be implemented in an offline manner. In these embodiments, the feature detection machine learning model may be generated in a first stage and provided as part of the feature detection machine learning module 218. In some embodiments, minor updates of the feature detection machine learning model may be implemented in an online manner. In such embodiments, an application that invokes the feature detection machine learning module 218 (e.g., the operating system 212, one or more of the other applications 214, etc.) may utilize the feature detections generated by the feature detection machine learning module 218, e.g., provide the feature detections to the user specific machine learning module 220, and generate a system log (e.g., actions taken by the user based on the feature detections, if permitted by the user, and results of further processing, if utilized as input for further processing). The system log may be generated periodically, e.g., hourly, monthly, quarterly, etc., and may be used to update the feature detection machine learning model, e.g., update the embeddings of the feature detection machine learning model, with the user's permission.

[0063] In some embodiments, the feature detection machine learning module 218 may be implemented in a manner that is adaptable to the particular configuration of the device 200 on which the feature detection machine learning module 218 executes. For example, the feature detection machine learning module 218 may determine a computation graph that utilizes available computational resources, e.g., the processor 202. For example, if the feature detection machine learning module 218 is implemented as a distributed application on multiple devices, the feature detection machine learning module 218 may determine the computations to be performed on each device in a manner that optimizes the computation. In another example, the feature detection machine learning module 218 may determine that the processor 202 includes a GPU with a particular number (e.g., 1000) of GPU cores and implement the feature detection machine learning module 218 accordingly (e.g., as 1000 individual processes or threads).

[0064] In some embodiments, the feature detection machine learning module 218 may implement an ensemble of trained models. For example, the feature detection machine learning model may include multiple trained models that are each applicable to the same input data. In these embodiments, the feature detection machine learning module 218 may select a particular trained model based on, for example, available computational resources, success rates with previous inferences, etc.

[0065] In some embodiments, the feature detection machine learning module 218 may execute multiple trained models. In these embodiments, the feature detection machine learning module 218 may combine outputs from the application of the individual models, for example, using a voting technique to score individual outputs from the application of each trained model, or by selecting one or more particular outputs. In some embodiments, such a selector is part of the model itself and serves as a connecting layer between the trained models. Furthermore, in these embodiments, the feature detection machine learning module 218 may apply a time threshold (e.g., 0.5 ms) for applying the individual trained models, and utilize only individual outputs that are available within this time threshold. Outputs that are not received within the time threshold may not be utilized, e.g., may be discarded. For example, such an approach may be implemented, for example, by the operating system 212 or one or more applications. 216 It may be preferable if there is a time limit specified between invocations of the feature detection machine learning module 218 .

[0066] User-specific machine learning modules 220 Example The user-specific machine learning module 220 generates a user-specific machine learning model that determines compression settings for the same input image analyzed by the feature detection machine learning module 218. In some embodiments, the user-specific machine learning module 220 includes a set of instructions executable by the processor 202 to generate the user-specific machine learning model. In some embodiments, the user-specific machine learning module 220 may be stored in the memory 204 of the device 200 and accessible and executable by the processor 202.

[0067] In some embodiments, the user-specific machine learning module 220 may use the training data to generate a trained model, specifically, a user-specific machine learning model. The training data may include any kind of data, such as user-specific features indicative of a user action with reference to one or more prior images for the feature detection machine learning model. For example, the user-specific features may indicate the degree to which the user is interested in an image. Indicating that an image is a favorite (e.g., marked as a favorite via explicit user input) may be considered an important image that should not be compressed. Other examples of user-specific features may include user actions, such as tagging other users in an image; sharing an image; creating a photo album or other image-based creation; commenting on an image; metadata that is important to the user, such as geolocation data indicating that the image was captured at an important location; downloading an image, editing an image, ordering prints of an image; editing an image; data that explicitly asks the user if the user likes an image, etc. In some embodiments, user actions may include actions on another user's image, such as using natural language processing to determine the sentiment of a comment on another user's image as a signal of interest in the image, indicating approval (e.g., liking) of another user's image, saving another user's image, downloading another user's image, etc. These user actions are signals of the value of the image and can be used as inputs to the model.

[0068] The training data may be obtained from any source, e.g., a data repository specifically marked for training, data for which permission is provided for use as training data for machine learning, etc. In embodiments in which one or more users authorize the use of their respective user data to train the machine learning model, the training data may include such user data. In embodiments in which users authorize the use of their respective user data, the data may include authorized data, such as images / videos or image / video metadata (e.g., images, corresponding features of the images, user-specific features associated with the user, descriptions of how the user-specific features indicate user actions with reference to one or more prior images, etc.), communications (e.g., messages on social networks; emails; chat data such as text messages, audio, video, etc.), documents (e.g., spreadsheets, text documents, presentations, etc.). In some embodiments, the prior images used by the feature detection machine learning module 218 to generate a feature detection machine learning model for how to identify one or more features are used by the user-specific machine learning module 220 to generate a user-specific machine learning model for determining compression settings based on the user-specific features indicative of user actions and the image features of each of the prior images.

[0069] For example, the trained model may be generated, for example, on a different device or provided as part of the image management application 103. In various embodiments, the trained model may be provided as a data file that includes a model structure or format (e.g., defining the number and type of neural network nodes, the connectivity between the nodes, and the organization of the nodes into multiple layers) and associated weights. The user-specific machine learning module 220 may read the data file for the trained model and implement a neural network with node connectivity, layers, and weights based on the model structure or format specified in the trained model.

[0070] The user-specific machine learning module 220 generates a trained model, referred to herein as a user-specific machine learning model. In some embodiments, the user-specific machine learning module 220 is configured to apply the user-specific machine learning model to data, such as data for the compression module 222, to identify compression settings for an input image. In some embodiments, the user-specific machine learning module 220 may include software code executed by the processor 202. In some embodiments, the user-specific machine learning module 220 may specify a circuit configuration (e.g., for a programmable processor, for a field programmable gate array (FPGA), etc.) that enables the processor 202 to apply the user-specific machine learning model. In some embodiments, the user-specific machine learning module 220 may include software instructions, hardware instructions, or a combination.

[0071] In some embodiments, the user-specific machine learning model may include one or more model formats or structures, for example, the model formats or structures may include any type of neural network, such as a linear network, a multi-layer feed-forward fully connected neural network, a CNN, or a deep neural network, such as a sequence-to-sequence neural network, as discussed in more detail above.

[0072] The model format or structure may specify the connectivity between various nodes and the organization of the nodes into layers. For example, the nodes of the first layer (e.g., input layer) may receive data as input data or application data 216. Such data may include, for example, one or more user-specific features per node, for example, if the feature detection machine learning model is used for analyzing user-specific features indicative of user actions associated with images. Subsequent intermediate layers may receive as input the output of the nodes of the preceding layer according to the connectivity specified in the model format or structure. These layers are sometimes referred to as hidden layers. The final layer (e.g., output layer) generates the output of the machine learning application. For example, the output may be a compression setting for the image based on the user-specific features. More specifically, the output may be a determination of the degree to which the user is interested in the image, which corresponds to a rating for the image that the user-specific machine learning module 220 maps to the compression setting. In some embodiments, the model format or structure also specifies the number and / or type of nodes in each layer.

[0073] The user-specific machine learning model receives input from the feature detection machine learning module 218 to identify features from the input image that is fed into the user-specific machine learning model. The input enables the feature detection machine learning model to identify features from the input image and determine whether the input image is interesting to the user. The user-specific machine learning model is trained on user-specific features indicative of user actions to identify which features in the image the user is interested in based on signals such as sharing a photo, viewing a photo, etc. In some embodiments, the signals are labels (e.g., user ranking, stars on a photo, at least one share, x number of views, etc.) to explicitly estimate the relative importance to the user. In some embodiments, these may be placed in a stack rank to create clusters. Within the cluster, the user-specific machine learning model uses a feature detection algorithm to generate a similarity metric, and then uses the similarity metric to estimate the relative importance to the user.

[0074] In some embodiments where there are no user-specific features available, the user-specific machine learning model may generate a baseline compression setting, for example, the baseline compression setting may be generated from user input from other users, where the input is a ranking of images based on blurriness or other type of metric of less interesting images.

[0075] The output of the user-specific machine learning model may be a rating that identifies the user's interest level, such as using a scale of 1-5, 1-10, etc., or a regression output (analog value), e.g., 7.83790. The user-specific machine learning model may map the rating to a compression setting, such as a compression ratio. Below is an example of how the rating may be mapped to a compression ratio. In this example, a rating of 1 would result in the compressed image occupying 0.2 of the original resolution of the original image.

[0076] [Table 1]

[0077] Although the above examples describe compression settings including ratings mapped to compression ratios, other examples of compression settings are possible. For example, the user-specific machine learning model may determine the compression technique, the image format (e.g., use of JPEG, WebP, HEIF, etc.), the parameters selected for optimization (e.g., dynamic range, image resolution, color, whether compression is progressive, etc.). In another example, the compression setting may be a decision to keep one or more features in the image at high resolution and compress the remainder of the image, such as by determining one or more regions of interest in the image. In another example, in addition to the user-specific machine learning model determining ratings for different features in the image, the user-specific machine learning model may also determine that quality trade-offs are considered for certain types of images. For example, dynamic range may be crucial for sunset images, while resolution may be more important for close-ups, etc.

[0078] In yet another example, the user-specific machine learning model may also indicate how to apply ratings when different features are included in the same image. For example, if an image typically has a rating of 5 if it contains a sunset and a rating of 2 if it contains food, the user-specific machine learning model may apply the rating that indicates the most interest, i.e., a rating of 5. In another embodiment, the user-specific machine learning model may determine that a particular user's response to an image suggests that ratings for multiple features should be averaged, that particular features should be associated with higher weights, etc.

[0079] In some embodiments, the user-specific machine learning model may determine that a rating of the input image meets an importance threshold, and in response to the rating meeting the importance threshold, the user-specific machine learning model provides a suggestion that the user share the input image. In another embodiment, in response to the rating meeting the importance threshold, the user-specific machine learning model prioritizes backing up the input image over other images associated with the user account that do not meet the importance threshold. For example, if the client device 115 is located in an area with limited Internet access, the user-specific machine learning model may instruct the client device 115 to send the image with the highest rating (or sorted in descending order based on the rating) to be forwarded to the server system 101 for storage. In another embodiment, in response to the rating meeting the importance threshold, the user-specific machine learning model instructs the user interface module 220 to provide a user interface including instructions for capture of a subsequent image if the scene depicted in the subsequent image has at least one of the one or more characteristics of the input image. For example, if the user is interested in taking photos of plants, the user interface may inform the user that water droplets on the photo may blur the petals. In another example, if a feature of interest is observed in an image, the camera can lock autofocus on the feature of interest (and allow the user to change focus with a tap).

[0080] In different embodiments, the user-specific machine learning model may include one or more models. One or more of the models may include multiple nodes arranged in layers per model structure or format. In some embodiments, the node may be, for example, a memoryless computational node configured to process one unit of input and generate one unit of output. The computation performed by the node may include, for example, multiplying each of the multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or intercept value to generate the node output. In some embodiments, the computation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a non-linear function.

[0081] In various embodiments, such computations may include operations such as matrix multiplication. In some embodiments, computations by multiple nodes may be performed in parallel, e.g., using multiple processor cores of a multi-core processor, using individual processing units of a GPU, or using dedicated neural circuitry. In some embodiments, a node may include memory, e.g., to store and use one or more previous inputs in processing a subsequent input. For example, a node with memory may include an LSTM node. An LSTM node may use memory to maintain state that allows the node to act like an FSM.

[0082] In some embodiments, the user-specific machine learning model may include embeddings or weights for individual nodes. For example, the user-specific machine learning model may be initiated as a number of nodes organized into layers as specified by the model format or structure. At initialization, a respective weight may be applied to each pair of nodes connected according to the model format, e.g., the connection between nodes in successive layers of a neural network. For example, the respective weights may be randomly assigned or initialized to a default value. The user-specific machine learning model may then be trained, e.g., using a user-specific training set, to generate results.

[0083] Training may include applying supervised learning techniques. In supervised learning, training data may include multiple inputs (e.g., ratings given to different types of images) and corresponding expected outputs (e.g., compression settings) for each input. Based on a comparison of the output (e.g., predicted ratings) of the feature detection machine learning model with the expected output (e.g., ratings provided by the user), the weight values ​​are automatically adjusted, for example, to increase the probability that the user-specific machine learning model will generate the expected output given similar inputs. An example of ratings provided by a user for categories of different types of images associated with the user is included below. In this example, the lowest rating is associated with the lowest importance and the highest rating is associated with the highest importance.

[0084] [Table 2]

[0085] In various embodiments, the trained model includes a set of weights or embeddings that correspond to the model structure. In embodiments where a training set of digital images is omitted, the user-specific machine learning module 220 may generate a user-specific machine learning model that is based on prior training, e.g., by a developer of the user-specific machine learning module 220, a third party, etc. In some embodiments, the user-specific machine learning model may include a set of weights that are fixed, e.g., downloaded from a server that provides the weights.

[0086] The user-specific machine learning module 220 may be implemented in different formats, in an offline manner and / or as an ensemble of trained models. Since this was described above with reference to the feature detection machine learning module 218, it will be understood that the same description would also apply to the user-specific machine learning module 220. Therefore, the description will not be repeated again.

[0087] Compression module 222 example The compression module 222 compresses the input image based on compression settings determined by the user-specific machine learning module 220. In some embodiments, the compression module 222 includes a set of instructions executable by the processor 202 to compress the input image. In some embodiments, the compression module 222 may be stored in the memory 204 of the device 200 and accessible and executable by the processor 202.

[0088] The compression module 222 may receive the input image from the feature detection machine learning module 218 and the compression settings from the user-specific machine learning module 220. The compression module 222 applies the compression settings to the input image. The compression module 222 may replace the original input image with a compressed input image to reduce the file size and more efficiently use the memory 204 and / or the storage device on which the image will be stored. In some embodiments, the compression module 222 may send the compressed input image to another location for storage. For example, if the image management application 103b is part of the client device 115a, the compression module 222 may send the compressed input image to the server system 101 for storage.

[0089] Example of User Interface Module 224 The user interface module 224 generates a user interface that receives input from a user. In some embodiments, the user interface module 224 includes a set of instructions executable by the processor 202 to compress an input image. In some embodiments, the user interface module 224 may be stored in the memory 204 of the device 200 and accessible and executable by the processor 202.

[0090] In some embodiments, the user interface module 224 generates a user interface for changing different settings associated with the image management application 103. In some embodiments, the user interface module 224 generates a user interface and receives user input to determine a baseline compression setting. For example, the user interface module 224 may generate a user interface viewable by a user associated with a user account.

[0091] The user interface may include two or more versions of the sample image, each of the images compressed with a different compression setting. The user interface may include a prompt requesting the user to identify a particular version of the sample image as a baseline compression setting for the user account. For example, the baseline compression setting may represent the lowest compression setting that the user is willing to accept for images associated with the user account. The user interface module 224 may request user input regarding the baseline compression setting multiple times to verify the accuracy of the baseline compression setting. For example, the user interface module 224 may provide images with different characteristics and display the user interface periodically (once a week, once a month, whenever there is a software update for the image management application 103), etc.

[0092] In some embodiments, the user interface module 224 may generate a warning in response to a user's selection. For example, if the user selects a high-resolution compression setting or no compression setting at all, the user interface module 224 may warn the user that storage space will be used up in a certain number of days or after a certain number of additional images are captured. This estimate may be based on the average photo size the user has uploaded in the last x days.

[0093] The user interface module 224 may send the user's selections to the user-specific machine learning module 220 for use in determining compression settings for the input image. The machine learning module 220 may then map the ratings to compression settings based on the baseline compression settings.

[0094] Any of the software in memory 204 may alternatively be stored on any other suitable memory location or computer readable medium. Additionally, memory 204 (and / or other connected storage device(s)) may store one or more messages, one or more taxonomies, electronic encyclopedias, dictionaries, thesauri, knowledge bases, message data, grammars, and / or other instructions and data used in the features described herein. Memory 204 and any other type of storage (such as magnetic disks, optical disks, magnetic tapes, or other tangible media) may be considered "storage" or "storage devices."

[0095] For ease of explanation, FIG. 2 illustrates a processor 202, a memory 204, an input / output interface 206, a camera 208, a display device 210, as well as software blocks 103, 218, and 220. ,2 A block is shown for each of 22 and 224. These blocks may represent one or more processors or processing circuits, operating systems, memory, input / output interfaces, applications, and / or software modules. In other embodiments, device 200 may not have all of the components shown and / or may have other elements, including other types of elements instead of or in addition to those shown herein. Although some components are described as performing the blocks and operations as described in some embodiments herein, environment 100, device 200, a similar system, or any suitable component or combination of components of such systems, or any suitable processor or processors associated with such systems may perform the blocks and operations described.

[0096] The methods described herein can be implemented by computer program instructions or code executable on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuits) and can be stored on a computer program product including a non-transitory computer-readable medium (e.g., storage medium), such as a semiconductor or solid-state memory, magnetic tape, removable computer diskette, random access memory (RAM), read-only memory (ROM), flash memory, magnetic hard disk, optical disk, solid-state memory drive, or other magnetic, optical, electromagnetic, or semiconductor storage medium. The program instructions can also be included in or provided as an electronic signal, for example, in the form of a software-as-a-service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods can be implemented in hardware (e.g., logic gates) or in a combination of hardware and software. Examples of hardware can be programmable processors (e.g., field programmable gate arrays (FPGAs), complex programmable logic devices), general-purpose processors, graphics processors, and application-specific integrated circuits (ASICs), etc. One or more of the methods may be implemented as part of or a component of an application executing on the system, or as an application or software running in conjunction with other applications and the operating system.

[0097] Example method FIG. 3 is a flow diagram illustrating an example method 300 of using a feature detection machine learning model 304 to identify one or more features 306 in an input image 302 and using a user-specific machine learning model 308 to determine compression settings 310 from the one or more features of the image, according to some embodiments.

[0098] The feature detection machine learning model 304 may include a deep neural network, such as a convolutional neural network (CNN), with a set of layers that build from pixels to more abstract objects. In some implementations, earlier layers of the CNN detect edges, and the deeper the layers, the more meaningful the human-defined features become. For example, intermediate layers may detect components of an object, and later layers may detect the object (or face) itself.

[0099] An input image 302 is provided as an input to the feature detection machine learning model 304. The input image 302 is from a user associated with a user account. The input image 302 may be received by an input layer of the set of layers. The input layer may be connected to a second layer of the set of layers. In some embodiments, one or more additional layers each receive an output of a previous layer as input and provide input to a next layer. The feature detection machine learning model 304 generates one or more features 306 based on the input image 302. The final layer of the set of layers may be an output layer. Continuing with this example, the output layer may output the one or more features 306.

[0100] In some embodiments, the output may include a corresponding probability that each of the features was correctly identified in the image. The output of the feature detection machine learning model 304 may be a vector of numbers, a probability value, or a set of probability values ​​(e.g., each corresponding to a particular stack of video frames). The output of the feature detection machine learning model 304 is provided as an input to the user-specific machine learning model 308.

[0101] The user-specific machine learning model 308 may also include a deep neural network, such as a CNN. In some embodiments, the user-specific machine learning model 308 may be generated using transfer learning by replacing the classification layer of the feature detection machine learning model 304 with a component that is trained with user-specific features associated with a user, where the user-specific features are indicative of user actions with reference to prior images. In some embodiments, the prior images are also used to train the feature detection machine learning model 304.

[0102] The user-specific machine learning model 308 may receive one or more features 306 via an input layer. The input layer may be connected to a second layer of the layers. In some embodiments, one or more additional layers may be included in the user-specific machine learning model 308, each receiving the output of the previous layer as input and providing input to the next layer. The last layer of the user-specific machine learning model 308 may be an output layer. In some implementations, the model 308 may have only one input layer that directly outputs the compression settings 310.

[0103] The user-specific machine learning model 308 may generate as output a compression setting 310 (prediction 310) and, optionally, a probability associated with the compression setting 310. In some embodiments, the compression setting 310 may include one or more ratings for one or more features in the input image 302. The probabilities may include a probability value, a set of probability values ​​(e.g., each corresponding to a rating for a particular feature in the input image 302), or a vector representation generated by the output layer of the user-specific machine learning model 308.

[0104] In some embodiments, the method 300 may be implemented on one or more of the client devices 115a, 115n, e.g., as part of the image management application 103b. In some embodiments, the method 300 may be implemented on the server device 104, e.g., as part of the image management application 103a. In some embodiments, the method 300 may be implemented on the server device 104 and on one or more of the client devices 115a, 115n.

[0105] In some embodiments, the method 300 may be implemented as software executable on a general purpose processor, e.g., a central processing unit (CPU) of a device. In some embodiments, the method 300 may be implemented as software executable on a special purpose processor, e.g., a graphics processing unit (GPU), a field programmable gate array (FPGA), a machine learning processor, etc. In some embodiments, the method 300 may be implemented as special purpose hardware, e.g., an application specific integrated circuit (ASIC).

[0106] FIG. 4 is a flow diagram illustrating an example of a method 400 for creating a training model, according to some embodiments.

[0107] The method 400 may begin at block 402. At block 402, it is determined whether user consent to the use of user data has been obtained. For example, the user interface module 224 may generate a user interface requesting a user for permission to use the user data in generating the feature detection machine learning model and / or the user-specific machine learning model. If user consent has not been obtained, at block 404, a baseline model is used instead of the feature detection machine learning model and / or the user-specific machine learning model.

[0108] If user consent is obtained, method 400 may proceed to block 406, where a training set of digital images and corresponding features are obtained. Block 406 may be followed by block 408. In block 408, a feature detection machine learning model is trained based on the training set and corresponding features, and after training, the feature detection machine learning model may identify image features in input images provided to the feature detection machine learning model. Block 408 may be followed by block 410.

[0109] At block 410, a training set of user-specific features associated with the user is obtained, the user-specific features being indicative of user actions with reference to one or more prior images. In some embodiments, the prior images are the same as a set of digital images. Block 410 may be followed by block 412. At block 412, a user-specific machine learning model is trained based on the user-specific features and the one or more prior images, and after training, the user-specific machine learning model determines a rating for an input image provided to the user-specific machine learning model.

[0110] FIG. 5 is a flow diagram illustrating an example of a method 500 for applying a model to an input image according to some embodiments.

[0111] At block 502, an input image associated with a user account is obtained. Block 502 may be followed by block 504. At block 504, one or more features of the input image are determined using a feature detection machine learning model. Block 504 may be followed by block 506. At block 506, compression settings in the input image are determined based on the one or more features in the input image using a user-specific machine learning model personalized to the user account. Block 506 may be followed by block 508. At block 508, the input image is compressed based on the compression settings.

[0112] Prior to training, each of the nodes may be assigned an initial weight and connections between nodes in different layers of the neural network may be initialized. Training may include adjusting the weights of one or more nodes and / or the connections between one or more pairs of nodes.

[0113] In some embodiments, a subset of the training set may be excluded in the initial training stage. This subset may be provided after the initial training stage, and the accuracy of the prediction (indication of whether to analyze the video) may be determined. If the accuracy falls below a threshold, further training may be performed with each additional digital image or user-specific feature from the training set to adjust the model parameters until the model accurately predicts their output.

[0114] Further training (second stage) may be repeated any number of times, e.g., until the model achieves a satisfactory level of accuracy. In some embodiments, the trained model may be further modified, e.g., compressed (to use fewer nodes or layers) or transformed (e.g., to be usable on different types of hardware). In some embodiments, different versions of the model may be provided, e.g., a client version of the model may be optimized for size or reduced computational complexity, while a server version of the model may be optimized for accuracy.

[0115] Although the methods 400, 450 have been described with reference to various blocks in Figures 4A and 4B, it will be understood that the techniques described in this disclosure can be performed without performing some of the blocks in Figures 4A and 4B. In some embodiments, one or more of the blocks shown in Figures 4A and 4B may be combined.

[0116] Additionally, although training has been described with reference to a training set, the feature detection machine learning model and the user-specific machine learning model may be trained on the fly. For example, if a user requests that an image be compressed using a particular compression setting, the feature detection machine learning model and the user-specific machine learning model may be updated to include the user information. In some embodiments, the user may manually provide annotations, e.g., providing a list of features and corresponding ratings for those features. With the user's permission, some embodiments may utilize such annotations to train the feature detection machine learning model and the user-specific machine learning model.

[0117] Although described with respect to specific embodiments thereof, these specific embodiments are merely exemplary and not limiting, and the concepts illustrated in the examples may be applied to other examples and embodiments.

[0118] In situations where certain embodiments discussed herein may collect or use personal information about a user (e.g., user data, information about the user's social network, the user's location and time at that location, the user's biometric information, the user's activity and demographic information), the user is provided with one or more opportunities to control whether information is collected, whether the personal information is stored, whether the personal information is used, and how information is collected, stored, and used about the user. That is, the systems and methods discussed herein collect, store, and / or use user personal information, particularly upon receiving explicit authorization to do so from the applicable user. Users have the ability to permanently delete these models.

[0119] For example, a user is provided with control over whether a program or feature collects user information related to that particular user or other users associated with the program or feature. Each user from whom personal information is to be collected is presented with one or more options that allow the user to control the collection of information related to that user and to give permission or approval as to whether and what portions of the information are collected. For example, a user may be provided with one or more such control options over a communications network. Additionally, certain data may be processed in one or more ways before being stored or used such that personally identifiable information is removed. As one example, the identity of a user may be treated such that personally identifiable information cannot be determined. As another example, the geographic location of a client device may be generalized to a larger area such that the specific location of the user cannot be determined.

[0120] It should be noted that the functional blocks, operations, features, methods, devices, and systems described in this disclosure may be combined or divided into different combinations of systems, devices, and functional blocks, as known to those skilled in the art. Any suitable programming language and programming techniques may be used to implement the routines of a particular embodiment. Different programming techniques, e.g., procedural or object-oriented techniques, may be employed. The routines may be executed on a single processing device or multiple processors. Although steps, operations, or computations may be presented in a particular order, the order may be changed in different particular embodiments. In some embodiments, multiple steps or operations shown as sequential in this specification may be performed simultaneously.

Claims

1. 1. A computer-implemented method comprising: Obtaining an input image associated with a user account; determining one or more features of the input image using a feature detection machine learning model; and a user-specific machine learning model personalized to the user account receives the one or more features to determine a compression setting for the input image, and outputs the compression setting and a rating corresponding to a degree of interest of a user associated with the user account in the one or more features in the input image; compressing the input image based on the compression setting; A computer-implemented method, wherein determining the compression setting includes mapping the rating to the compression setting.

2. The feature detection machine learning model is Obtaining a training set of digital images and corresponding features; and training the feature detection machine learning model based on the training set and the corresponding features, wherein after training, the feature detection machine learning model is capable of identifying one or more image features in the input image provided to the feature detection machine learning model.

3. 3. The computer-implemented method of claim 2, wherein the feature detection machine learning model includes a convolutional neural network (CNN) having multiple network layers, each network layer extracting the one or more image features at a different level of abstraction.

4. The user-specific machine learning model is obtaining a training set of user-specific features associated with a user that are indicative of user actions with reference to one or more prior images; and training the user-specific machine learning model based on the user-specific features and the one or more prior images, where after training, the user-specific machine learning model determines the rating of the input image provided to the user-specific machine learning model.

5. The computer-implemented method of claim 4 , wherein the training set further comprises image features for each of the one or more prior images.

6. 6. The computer-implemented method of claim 5, wherein the respective image features of the one or more preceding images are obtained by applying the feature detection machine learning model to the one or more preceding images.

7. providing a first user interface to a user associated with the user account having two or more versions of a sample image, each version compressed with a different compression setting; obtaining user input from the user identifying a particular version of the sample image; 2. The computer-implemented method of claim 1, further comprising selecting compression settings associated with the particular version of the sample image as baseline compression settings for the user account.

8. The computer-implemented method of claim 7, wherein the mapping is based on the baseline compression setting.

9. determining that the rating of the input image satisfies an importance threshold; In response to determining that the rating satisfies the importance threshold, providing a suggestion for the user to share the input image; prioritizing the backup of the input image over the backup of other images associated with the user account that do not meet the importance threshold; or 10. The computer-implemented method of claim 8, further comprising one or more of: providing a second user interface including instructions for capture of the subsequent image if the scene depicted in the subsequent image has at least one of the one or more characteristics of the input image.

10. 2. The computer-implemented method of claim 1, wherein determining the one or more features of the input image includes determining one or more regions of interest, and the compression setting includes a determination to keep the one or more features in the input image at high resolution and compress a remaining portion of the input image.

11. 1. A computing device comprising: A processor; and a memory storing instructions that, when executed by the processor, cause the processor to perform operations, the operations including: Obtaining an input image associated with a user account; determining one or more features of the input image using a feature detection machine learning model; and a user-specific machine learning model personalized to the user account receives the one or more features to determine a compression setting for the input image, and outputs the compression setting and a rating corresponding to a degree of interest of a user associated with the user account in the one or more features in the input image; compressing the input image based on the compression setting; The computing device, wherein determining the compression setting includes mapping the rating to the compression setting.

12. The feature detection machine learning model is Obtaining a training set of digital images and corresponding features; and training the feature detection machine learning model based on the training set and the corresponding features, where after training, the feature detection machine learning model is capable of identifying image features within the input image provided to the feature detection machine learning model.

13. The user-specific machine learning model is obtaining a training set of user-specific features associated with a user that are indicative of user actions with reference to one or more prior images; and training the user-specific machine learning model based on the user-specific features and the one or more prior images, where after training, the user-specific machine learning model determines the rating of the input image provided to the user-specific machine learning model.

14. The memory stores further instructions that, when executed by the processor, cause the processor to perform further operations, the further operations including: providing a first user interface to a user associated with the user account having two or more versions of a sample image, each version compressed with a different compression setting; obtaining user input from the user identifying a particular version of the sample image; and selecting a compression setting associated with the particular version of the sample image as a baseline compression setting for the user account.

15. The computing device of claim 14, wherein the mapping is based on the baseline compression setting.

16. A computer program causing one or more computers to carry out the method according to any one of claims 1 to 10.

Images