Image display with selective motion rendering
By determining a motion score for moving images and selectively displaying or storing them based on user preferences, the method optimizes resource usage in devices, addressing inefficient resource utilization in capturing and displaying moving images.
Patent Information
- Application Number
- JP2024021131
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-01-25
- Filing Date
- 2024-02-15
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2039-09-27
AI Technical Summary
Devices that capture and display moving images waste computational resources due to default settings that do not account for individual image characteristics or user preferences, leading to inefficient use of processor, memory, and storage capacity.
Implement a method to determine a motion score for moving images, displaying multiple frames if the score meets a threshold and converting to a still image if it does not, using a trained machine learning model to optimize resource usage based on user-specific preferences.
Reduces computational and storage requirements by selectively displaying or storing moving images, ensuring high-quality user experience while conserving resources.
Smart Images

Figure 0007583967000001 
Figure 0007583967000002 
Figure 0007583967000003
Abstract
Description
[Background technology]
[0001] background A user may capture images, including still images, such as photographs; images including subject movement, sometimes referred to as motion stills, motion photos, or live photos; 360-degree images; videos; and the like. For example, devices used to capture or view images, such as cameras, smartphones, wearable devices, computers, and the like, may have limited capacity to store images. Images including subject movement include multiple frames, which may require more storage capacity than photographs or still images. In such situations, a user must provide manual input regarding saving images to the local storage of the user device.
[0002] The background provided herein is intended to generally present the background of the present disclosure. The work of the inventors cited herein to the extent described in the Background section, and aspects of the description that may not qualify as prior art at the time of filing, are not admitted, expressly or impliedly, as prior art to the present disclosure. Summary of the Invention
[0003] overview Implementations described herein relate to methods, devices, and computer-readable media for displaying motion images including a plurality of frames. In some implementations, the computer-implemented method includes determining a motion score for the motion image based on the plurality of frames. The method further includes determining whether the motion score satisfies a display threshold. If the motion score satisfies the display threshold, the method further includes causing the motion image to be displayed with subject motion by successively displaying a plurality of frames of the motion image. If the motion score does not satisfy the display threshold, the method further includes causing the motion image to be displayed without subject motion by displaying a particular frame of the motion image.
[0004] Some embodiments include a computer-implemented method for displaying a motion image comprising a plurality of frames. The method may include determining a motion score for the motion image based on the plurality of frames of the motion image. The method may further include determining whether the motion score satisfies a display threshold. If the motion score satisfies the display threshold, the method may further include causing the motion image to be displayed with subject motion by successively displaying the plurality of frames of the motion image. If the motion score does not satisfy the display threshold, the method may further include causing the motion image to be displayed without subject motion by displaying a particular frame of the plurality of frames of the motion image.
[0005] In some implementations, the motion image is displayed in a user interface that includes the one or more other images, in which a size of the motion image in the user interface is larger than at least one of the one or more other images if the motion score meets a display threshold.
[0006] In some implementations, the method may include determining a respective motion score for the one or more other images. The method may further include selecting a display threshold based on the respective motion scores. The selecting step includes determining a number of the one or more other images each associated with a motion score higher than the motion score of the video image. If the number is equal to or greater than a threshold number, the selecting step includes setting the display threshold to a number higher than the motion score of the video image. If the number is less than the threshold number, the selecting step further includes setting the display threshold lower than the motion score of the motion image. Some implementations further include determining the threshold number based at least in part on a plurality of execution threads of a software application that displays the motion image.
[0007] In some implementations, determining the motion score includes determining a degree of stability of a background portion of the video, a type of object in the video, and a degree of object movement in the video based on pixel values of each of the plurality of frames, and calculating the motion score based on the degree of stability, the type of object, and the degree of object movement. In some implementations, if the type of object is landscape or inanimate object, calculating the motion score includes setting the motion score to a value below a display threshold.
[0008] In some implementations, calculating the motion score includes determining whether an extent of object movement meets a threshold level of movement. If it is determined that the extent of object movement meets the threshold level of movement, calculating the motion score further includes setting the motion score to a value that meets a display threshold. If it is determined that the extent of object movement does not meet the threshold level of movement, calculating the motion score further includes setting the motion score to a value that does not meet a display threshold. In some implementations, the first type of object is a face and the second type of object is a body, and a first threshold level of movement associated with a face is lower than a second threshold level of movement associated with a body.
[0009] In some implementations, determining the motion score for the video may further include setting the motion score to a value that meets a display threshold if the number of views of the video meets a viewer count threshold or if at least one motion artifact has been created based on the video.
[0010] In some implementations, determining the motion score for the video includes applying a trained machine learning model comprised of a neural network including a plurality of nodes organized into a plurality of layers and configured to determine the motion score, In these implementations, the video is provided as an input to the trained machine learning model.
[0011] In some implementations, the video images are captured by the client device and stored in a volatile memory of the client device. In these implementations, the method may further include determining that a motion score satisfies a storage threshold. If the motion score satisfies the storage threshold, the method may further include storing the video images in a non-volatile memory of the client device. If the motion score does not satisfy the storage threshold, the method may further include automatically changing the video images to a still image that includes only certain frames of the plurality of frames and has a smaller file size than the video images. In some implementations, changing the video images to a still image includes deleting other frames of the plurality of frames and storing the still image in a non-volatile memory of the client device.
[0012] Some implementations may include a device for displaying motion images comprised of a plurality of frames. The device may include a processor and a non-transitory computer-readable medium having instructions stored thereon. The instructions, when executed by the processor, cause the processor to perform operations that may include determining a motion score for the motion image based on the plurality of frames. The operations may further include determining whether the motion score satisfies a display threshold. If the motion score satisfies the display threshold, the operations may display the motion image with subject motion by successively displaying a plurality of frames of the motion image. If the motion score does not meet the display threshold, the operations may further include causing the motion image to be displayed without subject motion by displaying a particular frame of the plurality of frames of the motion image.
[0013] In some implementations, determining a motion score for the video includes applying a pre-trained machine learning model stored on the device, the pre-trained machine learning model comprising a neural network including a plurality of nodes organized into a plurality of layers and configured to determine the motion score, In some implementations, the video is provided as an input to the pre-trained machine learning model.
[0014] In some implementations, the operations may further include determining that a number of views of the video meets an audience threshold or that motion artifacts have been created from the video. The operations may further include updating the trained machine learning model in response to the determination. In some implementations, the updating the trained machine learning model includes adjusting a weight of at least one node of the plurality of nodes. In some implementations, the updating the trained machine learning model includes updating a connection between at least one pair of nodes of the plurality of nodes.
[0015] In some implementations, the operations may further include determining a degree of stability of a background portion of the video and determining a type of subject in the video and a degree of subject movement in the video. In some implementations, the degree of stability of the background portion, the type of subject, and the degree of subject movement are provided as further inputs to the trained machine learning model.
[0016] In some implementations, the operations may further include accepting a user input to display the motion image as a still image by displaying only a particular frame of the plurality of frames. The operations may further include updating the trained machine learning model in response to the user input. In some implementations, the operation of updating the machine learning model includes adjusting a weight of a node of the plurality of nodes. In some implementations, the operation of updating the machine learning model includes at least one of the operations of updating a connection between at least one pair of nodes of the plurality of nodes.
[0017] In some implementations, the device may further include a camera that captures video images. In some implementations, the video images are stored in a volatile memory of the device. In these implementations, the operations may include determining whether a motion score meets a storage threshold. If the motion score meets the storage threshold, the operations may further include storing the video images in a non-volatile memory of the device. If the motion score does not meet the storage threshold, the operations may include automatically changing the video images to a still image that includes only certain frames of the plurality of frames and has a smaller file size than the video images. In some implementations, changing the video images to a still image may include deleting other frames of the plurality of frames and storing the still image in a non-volatile memory of the device.
[0018] Some implementations may include a non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform operations that may include determining a motion score for the video based on a plurality of frames of the video. The operations may further include determining whether the motion score satisfies a display threshold. If the motion score satisfies the display threshold, the operations may further include displaying the video with subject motion by successively displaying a plurality of frames of the video. If the motion score does not satisfy the display threshold, the operations may include displaying a plurality of frames of the video with subject motion by successively displaying a plurality of frames of the video. The method may further include displaying a particular one of the frames, thereby displaying the moving image without subject motion.
[0019] In some implementations, the non-transitory computer-readable medium may store further instructions that, when executed by the processor, cause the processor to perform operations that may include determining a degree of stability of a background portion of the motion image and determining a type of object in the motion image and a degree of object movement in the motion image. In some implementations, the motion score is based on the degree of stability, the type of object, and the degree of object movement.
[0020] In some implementations, the non-transitory computer-readable medium may store further instructions that, when executed by the processor, cause the processor to perform operations that may include determining a respective motion score for the one or more other images and selecting a display threshold based on the respective motion scores. In some implementations, the selecting operation may include determining a number of the one or more other images each associated with a motion score higher than the motion score of the video image. If the number is greater than or equal to a threshold number, the operation further includes setting the display threshold higher than the motion score of the video image. If the number is less than the threshold number, the operation further includes setting the display threshold lower than the motion score of the video image. [Brief description of the drawings]
[0021] [Figure 1] FIG. 1 is a block diagram of an exemplary network environment that may be used for one or more implementations described herein. [Diagram 2] 1 is a flow diagram illustrating an exemplary method for displaying and / or storing motion images according to some implementations. [Diagram 3] 1 is a flow diagram illustrating an example method for determining a motion score for a motion image according to some implementations. [Figure 4] 4 is a flow diagram illustrating another example method for determining a motion score for a motion image according to some implementations. [Diagram 5] FIG. 1 illustrates an exemplary motion picture. [Figure 6] 1 is an illustration of an exemplary user interface including an image grid according to some embodiments. [Figure 7] FIG. 1 is a block diagram of an example device that may be used for one or more implementations described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0022] Detailed Description Some implementations include methods, devices, and computer-readable media having instructions for displaying motion images that include a plurality of frames. For example, the motion images may include a plurality of frames captured by a camera over a short period of time, e.g., three seconds or less. The motion images may be captured without significant camera movement, e.g., without panning, zooming, tilting, or other actions that require camera movement.
[0023] The video images may depict subjects and backgrounds. The video images may have different degrees of interest to the user, for example, a video image depicting a smiling friend, a jumping pet, a dance move, a sports move, etc. may be of high interest to the user. In another example, a video image depicting little or no movement between frames, a video image not including a subject of interest to the user, such as an image of a document or an inanimate object, may be of low interest to the user.
[0024] A technical problem is that devices capable of capturing, displaying, or storing video waste computational resources in such operations. For example, many devices provide default settings for capturing video. Such devices can capture and store video, but the device's camera must be operated and the resulting image data must be stored on the device. In this case, even if the user experience is not improved when the video is displayed, the device's resources are utilized to capture and store the video, wasting computational capacity to capture and process the video and storage capacity to store the video. For example, a displayed video provides a high quality user experience when there is significant subject movement in the image that can be recognized by a viewing user. Furthermore, when a user views such a video, the device's processor, memory, and display screen are utilized to load multiple frames to display the video depicting the subject movement. A user can selectively disable the capture, storage, or display of video, but such settings are generic in nature and do not take into account the characteristics of individual images or the user's preferences.
[0025] The implementations described herein automatically determine a motion score for a video by analyzing multiple frames of the video. If the motion score for the video meets a display threshold, the multiple frames depicting the subject's motion are displayed in a user interface. Otherwise, certain frames of the video are displayed and still images are provided. Such selective display can reduce the use of computational resources utilized to display the video.
[0026] In some implementations, a user interface is provided that displays multiple images. In these implementations, the motion score is evaluated based on, for example, available hardware capacity, and only a limited number of images are displayed as a video. For example, the number of images may be based on the number of threads of the program or software application generating the user interface that can run on the available hardware. The user interface is thus rendered to the available hardware resources, which may allow different types of devices to display the user interface without having to modify the implementation code or modify the video. Furthermore, in some implementations, the size of the video in the user interface is determined based on the motion score of the video. Such customization of image size may provide a user interface that is easier to use. For example, an image grid that includes images of different sizes may display certain images at a larger size than other images, e.g., allowing a user to appreciate such images in more detail since a larger portion of the available display screen area is utilized for such images. Such a display may allow a user to make decisions about whether to edit an image, view an image full screen, create an image-based work, share an image, etc., with more confidence than if the images were displayed at a smaller size. Selectively displaying certain images at a larger size also ensures that any number of images can be displayed on the screen while still displaying sufficient detail.
[0027] Furthermore, in some implementations, the motion score is compared to a storage threshold and the motion image is converted to a still image (including one frame) before storage. Such selective conversion can reduce the storage capacity required to store the motion image. If user images are uploaded to a server, selective conversion can reduce the transmission cost of sending the images to the server and can reduce the utilization of server storage capacity for storing the motion image.
[0028] In yet another embodiment, a trained machine learning model is utilized to determine the motion score. A trained ML model may provide a computationally efficient mechanism for determining the motion score because the model is optimized for this application and can be implemented on application-specific hardware, such as an FPGA, an ASIC, a GPU, or a machine learning processor.
[0029] In some implementations where a trained ML model is utilized, with user permission, the model updates based on user interactions, e.g., viewing video, generating motion artifacts, discarding motion components (frames) from video, etc. Because the motion scores determined by the model are user specific, updates to such models can further improve video display and storage. Furthermore, with user permission, such model updates may be accepted from multiple users and used to further train the ML model, e.g., in a federated learning approach, to improve motion score determination.
[0030] In some implementations where a motion score is determined during image capture and prior to storage in non-volatile memory, the described techniques can reduce non-volatile memory utilization by discarding frames of video that may not be of interest to the user. In some implementations, the techniques described herein provide a user interface that allows a user to selectively discard motion, thereby reducing resource usage compared to the default mode of video capture.
[0031] The same reference numbers are used in FIG. 1 and other drawings to identify the same elements. Letters following a reference number, such as "156a," indicate that the text is specifically referring to the element having that particular reference number. A reference number in text without a following letter, such as "156," refers to any or all of the elements having that reference number in the drawings (e.g., "156" in the text refers to reference numbers "156a" and / or "156b" in the drawings).
[0032] FIG. 1 illustrates a block diagram of an exemplary network environment 100 that may be used in some embodiments described herein. In some embodiments, the network environment 100 includes one or more server systems, such as server system 102 in the example of FIG. 1. The server system may be in communication with a network 130, for example. The server system 102 may include a server device 104 and a database 106 or other storage device. The database 106 may store one or more images and metadata associated with the one or more images. In some embodiments, the server device 104 may provide an image management application 156b. The image management application 156b may access images stored in the database 106.
[0033] The network environment 100 may also include one or more client devices, e.g., client devices 120, 122, 124, and 126. The client devices 120, 122, 124, and 126 may communicate with each other and / or with the server system 102 and / or the second server system 140 via a network 130. The network 130 may be any type of communication network, including one or more of the Internet, a local area network (LAN), a wireless network, a switched or hubbed connection, and the like. In some implementations, the network 130 may include peer-to-peer communication between devices, for example, using a peer-to-peer wireless protocol (e.g., Bluetooth, Wi-Fi Direct, and the like). An example of peer-to-peer communication between two client devices 120 and 122 is illustrated by arrow 132.
[0034] For ease of explanation, FIG. 1 shows one block for the server system 102, the server device 104, and the database 106, and four blocks for the client devices 120, 122, 124, and 126. The server block 102 may represent multiple systems, multiple server devices, and multiple network databases, and the block may be provided in a different configuration than that shown. For example, the server system 102 may represent multiple server systems that may communicate with other server systems over the network 130. In some implementations, the server system 102 may include, for example, a cloud hosting server. In some examples, the database 106 and / or other storage may be provided in a server system block(s) separate from the server device 104, and may communicate with the server device 104 and other server systems over the network 130. There may also be any number of client devices. Each client device may be any type of electronic device, such as a desktop computer, a laptop computer, a portable or mobile device, a mobile phone, a smartphone, a tablet computer, a television, a TV set-top box or entertainment device, a wearable device (e.g., display glasses or goggles, a watch, a headset, an armband, an accessory, etc.), a personal digital assistant (PDA), a media player, a gaming device, etc. Some client devices may also have a local database similar to database 106 or other storage. In some implementations, network environment 100 may not have all of the components shown and / or may have other elements, including other types of elements, instead of or in addition to the components described herein.
[0035] In various implementations, end users U1, U2, U3, and U4 may communicate with server system 102 and / or with each other using respective client devices 120, 122, 124, and 126. In some examples, users U1, U2, U3, and U4 may interact with each other via applications running on respective client devices and / or server system 102 or second server system 140, and / or via network services, such as social network services or other types of network services, implemented on server system 102. For example, respective client devices 120, 122, 124, and 126 may communicate data to or from one or more server systems (e.g., system 102, second server system 140).
[0036] In some implementations, the server system 102 may provide appropriate data to the client devices to enable each client device to receive communication or shared content uploaded to the server system 102 and / or the network service. In some examples, the users U1-U4 may interact via audio or video conferences, audio, video, or text chat, or other communication modes or applications. The network service implemented by the server system 102 may include systems that enable users to communicate, link and associate, upload and post shared content, such as images, text, video, audio, and other types of content, and / or perform other functions. For example, a client device may display received data, such as content posts that have been sent or streamed to the client device and sent from a different client device through the server and / or network service (or directly from a different client device) or sent from the server system and / or network service. In some implementations, the client devices may communicate with each other directly, for example, using peer-to-peer communication between client devices as described above. In some implementations, the client devices may communicate with each other directly, such as using peer-to-peer communication between client devices as described above. In this regard, a "user" may include one or more programs or one or more virtual entities, as well as a person who interacts with a system or network.
[0037] In some implementations, any of client devices 120, 122, 124, and / or 126 may provide one or more applications. For example, as shown in FIG. 1, client device 120 may provide a camera application 152 and an image management application 156a. Client devices 122-126 may also provide similar applications. For example, camera application 152 may provide users of each client device (e.g., users U1-U4) with the ability to take images using the camera of their respective user device. For example, camera application 152 may be a software application that executes on client device 120.
[0038] In some implementations, camera application 152 may provide a user interface. For example, the user interface may enable a user of client device 120 to select an image capture mode, such as a still image (or photo) mode, a burst mode (e.g., taking multiple images in succession over a short period of time), a motion picture mode, a video mode, a High Dynamic Range (HDR) mode, etc. For example, a motion picture mode may correspond to capturing a still image upon activation of the camera of client device 120, and capturing a short clip that includes multiple frames before and after the camera is activated. For example, the duration of the short clip may be 1-3 seconds.
[0039] In some implementations, the motion image may be captured in the volatile memory of the client device 120 as one image file with still images in the image data portion and multiple frames in a second portion, e.g., image metadata portion. In some implementations, the motion image may be captured in the volatile memory of the client device 120 as two separate image files, a first file with still images and a second file with short clips. In some implementations, the first file and / or the second file may include image metadata indicating that the other file was taken at the same time. In some implementations, the camera application 152 may implement the methods described herein with reference to Figures 2-4. In some implementations, the image management application 156a and / or the image management application 156b may implement the methods described herein with reference to Figures 2-4.
[0040] The camera application 152 and the image management application 156a may be implemented using hardware and / or software of the client device 120. In different implementations, the image management application 156a may be a standalone application running on, for example, any of the client devices 120-124, or may operate in conjunction with an image management application 156b provided on the server system 102. With user permission, the image management application 156 may perform one or more automatic functions, such as backing up images (e.g., to a database 106 of the server system 102), enhancing images, stabilizing images, recognizing one or more objects (e.g., faces, bodies, inanimate objects, etc.) contained in the images, etc. Image stabilization may be based on input from an accelerometer, gyroscope, or other sensors of the client device 120, and / or based on a comparison of multiple frames of a moving image or video.
[0041] The image management application 156 provides functions for displaying images in a user interface (e.g., in a one-up view with a single image, a grid view with multiple images, etc.), editing images (e.g., adjusting image settings, applying filters, changing the focus of an image, etc.), and The client devices 122-126 may also provide image management functionality, such as the ability to view, edit, or delete one or more frames of a video or movie, share images with other users (e.g., at client devices 122-126), archive images (e.g., store images so that they do not appear in the primary user interface), and create image-based works (e.g., motion-based artifacts such as collages, photo books, animations, stories, video loops, etc.).
[0042] User interfaces on client devices 120, 122, 124, and / or 126 may enable the display of user and other content, including images, videos, data, and other content, as well as communications, privacy settings, notifications, and other data. Such user interfaces may be displayed using software on the client device, software on the server device, and / or a combination of client and server software running on server device 104, e.g., application software or client software in communication with server system 102. The user interfaces may be displayed on a display device of the client device or server device, e.g., a touch screen or other display screen, a projector, etc. In some implementations, application programs running on the server system may communicate with the client devices to receive user input at the client devices and output data, such as image data, audio data, etc., at the client devices.
[0043] In some implementations, the server system 102 and / or any of the one or more client devices 120-126 may provide a communications application program. The communications program may enable a system (e.g., a client device or a server system) to provide options for communicating with other devices. The communications program may provide one or more associated user interfaces for display on a display device associated with the server system or the client device. The user interface may provide the user with various options for selecting a communication mode, a user or device to communicate with, etc. In some examples, the communications program may provide an option for broadcasting a content post to a broadcast area and / or output a notification that a content post has been received by the device, that the device is in a defined broadcast area for the post. The communications program may display or output the sent and received content posts, for example, in various formats. The content posts may include, for example, images shared with other users.
[0044] Other implementations of the features described herein can utilize any type of system and / or service. For example, other network-connected (e.g., Internet-connected) services can be used instead of or in addition to social networking services. Any type of electronic device can utilize the features described herein. Some implementations can provide one or more features described herein on one or more client or server devices that are disconnected from or intermittently connected to a computer network. In some examples, a client device that includes a display device or has a display device connected thereto can display content posts that are stored on a storage device local to the client device, e.g., previously received over a communication network.
[0045] FIG. 2 is an illustrative diagram for displaying and / or storing video images according to some embodiments. 1 is a flow diagram illustrating a typical method 200. In some implementations, the method 200 may be implemented, for example, on the server system 102 shown in FIG. 1. In some implementations, some or all of the method 200 may be implemented on one or more client devices 120, 122, 124, or 126 shown in FIG. 1, on one or more server devices, and / or on both the server device(s) and the client device(s). In the described example, the implementation system includes one or more digital processors or processing circuits ("processors") and one or more storage devices (e.g., database 106 or other storage). In some implementations, different components of one or more servers and / or one or more clients may perform different blocks or other portions of the method 200. In some examples, a first device is described as performing blocks of the method 200. Some implementations may have one or more blocks of the method 200 performed by one or more other devices (e.g., other client devices or server devices) that may send results or data to the first device.
[0046] In some implementations, method 200 or portions thereof may be automatically initiated by a system. In some implementations, the implementation system is the first device. For example, the method (or portions thereof) may be executed periodically or based on one or more specific events or conditions, such as, for example, an application (e.g., camera application 152, image management application 156, etc., being initiated by a user), the receipt of one or more images newly uploaded to or accessible by the system, a predetermined amount of time having elapsed since method 200 was last executed, and / or the occurrence of one or more other conditions that may be specified in the settings that the method loads. In some implementations, such conditions may be specified by a user in the user's stored custom preferences.
[0047] In various implementations, client device 120 may be a standalone camera, another device with a camera (e.g., a smartphone, a tablet, a computer, a wearable device such as a smart watch, a headset, etc.), or other client device capable of receiving images captured by another device. In some implementations, client device 120 may be a capture-only device, e.g., a camera without a screen. In some implementations, client device 120 may be a display-only device, e.g., a device with a screen on which images can be displayed, but without camera functionality or other functionality for capturing images. In some implementations, client device 120 may have both image capture and image display capabilities.
[0048] In some implementations, client device 120 may include one camera for capturing images. In some implementations, client device 120 may include multiple cameras (or lenses). For example, a smartphone or other device may include one or more front-facing cameras (on the same side as the device's screen) and / or one or more rear-facing cameras. In some implementations, the one or more front-facing or rear-facing cameras may work in tandem during image capture, e.g., a first camera may capture depth information and a second camera may capture image pixels. In some implementations, different cameras (e.g., telephoto lenses, wide-angle lenses, etc.) may be used for different types of image capture, e.g., with different zoom levels. In some implementations, client device 120 may be configured to capture 360-degree images. In some implementations, a camera or lens may capture images using one image sensor (e.g., a CCD or CMOS sensor) or multiple sensors. In some implementations, other sensors, e.g., depth sensors, may be used in conjunction with one or more cameras during image capture.
[0049] In some implementations, client device 120 may combine raw image data captured by one or more camera (or lens) image sensors with other data obtained from other sensors (e.g., accelerometers, gyroscopes, position sensors, depth sensors, etc.) to form an image. For example, when client device 120 is operated in a mode that captures multiple image frames (e.g., burst or motion mode in which multiple frames are captured in rapid succession to create a video image, video mode in which a video is captured, high dynamic range mode in which multiple images of different exposures are combined into a single composite image, etc.), data obtained from the sensors may be used to stabilize the captured images. For example, the captured frames may be aligned using accelerometer or gyroscope data to compensate for camera movement caused by a user's shaking hands while capturing the images. In some implementations, the captured images or videos may be cropped to create stabilized images, e.g., images or videos with reduced background motion.
[0050] Client device 120 allows a user to capture images in different modes, such as, for example, a still image (or photo) mode for capturing a single frame, a burst or video mode for capturing multiple frames, a video mode for capturing video, etc. In some implementations, method 200 may be performed when a camera captures an image that comprises multiple frames.
[0051] Client device 120 may allow a user to view images, such as images captured by client device 120 or other images associated with the user, in different user interfaces. For example, a one-up or slideshow mode may be provided that allows a user to view one image at a time. In another example, a gallery mode may be provided that allows a user to view multiple images simultaneously, for example as an image grid.
[0052] In some implementations, the client device 120 may perform the method 200. In another example, a client device or a server device may perform the method 200. In some implementations, the method 200 may be initiated based on user input. A user (e.g., an operator or an end user) may, for example, select to initiate the method 200 from a display user interface, such as an application user interface or other user interface. In some implementations, the method 200 may be implemented by a server device.
[0053] In some implementations, method 200 may be initiated automatically, for example, when a user of a client device operates a camera to capture video images, or when the user of the client device views images stored on the client device or on a server device, etc.
[0054] An image as referred to herein may include a digital image having pixels with one or more pixel values (e.g., color values, brightness values, etc.). An image may be a still image (e.g., a still photograph, an image having one frame, etc.) or a moving image (e.g., an image including multiple frames, such as an animation, animated GIF, a cinemagraph where some parts of the image include movement and other parts are still, a moving image including multiple frames with or without sound).
[0055] Block 202 verifies whether user consent (e.g., user permission) has been obtained to utilize user data in implementing method 200. The data may include images taken by a user using a client device, e.g., images stored or accessed by a user using a client device, image metadata, user data related to usage of messaging applications, user preferences, user biometric information, user characteristics (e.g., identity, name, age, gender, occupation, etc.), information about a user's social network and contacts, social or other types of behavior and activity, user-created or written content, ratings, and opinions, a user's current location, historical user data, images generated, received, and / or accessed by a user, images viewed or shared by a user, etc. One or more blocks of the methods described herein may use such user data in some embodiments.
[0056] If user consent has been obtained from an associated user whose user data may be used in method 200, then at block 204 it is determined that the blocks of the method herein may be performed with the user data available as described above, and the method continues to block 212. If user consent has not been obtained, then at block 206 it is determined that the blocks are to be performed without utilizing user data, and the method continues to block 212. In some embodiments, if user consent has not been obtained, then the blocks are performed without utilizing user data, and utilizing synthetic data and / or generic or commonly accepted and commonly available data. In some embodiments, if user consent has not been obtained, then method 200 is not performed.
[0057] At block 212, a motion score for the video is determined. In some implementations, method 200 may be performed when a new video is captured by a camera. In these implementations, the determination of the motion score for the video may be based on image data (e.g., pixel values of multiple frames of the video) and image metadata. In some implementations, method 200 may be performed to determine a motion score for a stored video (e.g., stored in non-volatile memory of client device 120) that includes multiple frames and for which a motion score is determined. For example, the motion score may be determined when the image metadata or other data associated with the video (e.g., stored in a database of images) does not include a motion score or includes an expired motion score. For example, the motion score for a video may expire after a certain number of days have passed since the video was captured or last viewed by a user (e.g., of client device 120), when the video is modified, when an animated GIF, video, collage, or other work based on the video is generated, etc.
[0058] Determining the motion score based on multiple frames of the video may be performed, for example, using the method 300 described with reference to Figure 3 or the method 400 described with reference to Figure 4. For example, if the video is associated with a valid motion score that has been pre-calculated and stored in the image metadata or database of images, determining the motion score may include accessing the image metadata or database of images to retrieve the motion score. Block 212 may be followed by block 214.
[0059] At block 214, it is determined whether the motion score of the video satisfies a display threshold. With user permission, the value of the display threshold may be determined based on user data, such as the user's previous views of a user interface depicting the video, manual toggling of an image capture / display mode with or without subject motion, etc. In some implementations, for example, when video is displayed as part of a user interface that includes one or more other images, the display threshold may be set based on a target number of video images to display at one time such that more than the target number of video images are not displayed with motion turned on (e.g., by displaying multiple frames sequentially). In some implementations, for example, when video is displayed as a single image, the display threshold may be set based on a target number of video images to display at one time such that more than the target number of video images are not displayed with motion turned on (e.g., by displaying multiple frames sequentially). When displayed in an image mode, e.g., one-take mode or as part of a slide show, the display threshold may be set to a value determined based on interest in subject motion within multiple frames of the motion image when displayed sequentially. For example, interest in subject motion may be based on similarity of subject motion in the motion image to subject motion in other motion images viewed by the user (or other users) where motion was displayed in one-take mode or slide show mode.
[0060] In some implementations, for example, when a motion image is displayed as part of a user interface that includes one or more other images (e.g., as part of an image grid), the display threshold may be set to a value based on the motion score of the one or more other images. For example, where the one or more other images may include other motion images. If at least a threshold number of the other images have a higher motion score than the motion image, the display threshold may be set to a value such that the motion score of the motion image does not meet the display threshold. For example, the display threshold may be set to a value higher than the motion score of the motion image. If the number of other images that have a higher motion score than the motion image is less than a threshold number, the display threshold may be set to a value such that the motion score of the motion image meets the display threshold. For example, the display threshold may be set to a value lower than the motion score of the motion image.
[0061] In some implementations, the threshold number may be based on the hardware capabilities of a device executing method 200, e.g., client device 120. For example, method 200 may be executed (or called) by a software application for displaying video footage. The software application may be multi-threaded, e.g., with one or more execution threads of the software application running on a hardware processor of client device 120. In some implementations, each individual execution thread may process a corresponding video footage to be displayed in a grid and render multiple frames for display. In these implementations, the threshold number may be based on the number of execution threads, e.g., the threshold number may be less than or equal to the number of execution threads.
[0062] In some implementations, the number of execution threads may be based on hardware capabilities, such as, for example, the number of processing cores of a hardware processor of client device 120, available memory of client device 120, etc. For example, if each execution thread runs on a corresponding processing core, the number of execution threads may not exceed the number of available processing cores. Thus, the number of available processing cores may be based on available power (e.g., battery), operating system settings (e.g., configuration settings that limit the number of execution threads per application), etc.
[0063] Selection of the threshold number based on hardware capabilities ensures that the hardware can support the number of moving images rendered for display in a user interface, e.g., an image grid. For example, the threshold number may be set to 1 such that the image grid contains only one image that contains multiple frames (e.g., if there is only one processing core available to render the image in the image grid). In this example, other moving images included in the motion grid are displayed as still images by rendering one frame of each of the other moving images for display. In another example, if four cores are available to render images included in the motion grid, the threshold number may be set to 4. Setting the threshold number in this manner provides the technical advantage of rendering the image grid without delay using available hardware capabilities.
[0064] If the motion score of the video meets the display threshold, block 214 is followed by block 216. If the motion score of the video does not meet the display threshold, block 214 is followed by block 216. This is followed by block 218.
[0065] At block 216, the motion image is displayed with the subject's movement. The motion image may be displayed, for example, as part of a user interface provided by an image gallery application. In some implementations, displaying the motion image includes, for example, displaying multiple frames of the motion image in sequence such that the subject's movement within the multiple frames of the motion image is perceptible by a user. For example, the motion image may be displayed in the user interface in a one-up mode or slideshow mode in which one image is displayed at a time.
[0066] In some implementations, the step of displaying the motion image further includes displaying the motion image in a user interface including an image grid including a plurality of images at a larger size than the other images. Selection of the threshold number depending on the circumstances described with reference allows displaying the image grid having one or more motion images associated with a motion score that meets the display threshold and displayed with subject motion. In some implementations, the motion images associated with a motion score that meets the display threshold may be displayed in the user interface at a larger size than one or more other images in the grid. Other motion images in the user interface, each associated with a score that does not meet the display threshold, are displayed in the user interface without subject motion, for example by displaying only certain frames of such motion images. Block 216 may be followed by block 220.
[0067] If the motion score of the video does not meet the display threshold, block 218 is executed to display the video without subject motion, for example, by displaying only certain frames of the video. Displaying certain frames in this manner may conserve computational resources since other frames of the video will not be displayed. Block 218 may be followed by block 220.
[0068] At block 220, it is determined whether the motion score satisfies a storage threshold. The storage threshold may be based on various factors. For example, the storage threshold may be based on the available non-volatile memory capacity of the user device, e.g., client device 120, that stores the video. In another example, for example, if the video is stored on a server, the storage threshold may be based on the server's storage capacity allocated to the particular user account that stores the video. In some implementations, the storage threshold may be set based on a combination of the user device's non-volatile memory capacity and the server's allocated storage capacity. If the motion score satisfies the storage threshold, block 220 is followed by block 222. If the motion score does not satisfy the storage threshold, block 220 may be followed by block 224.
[0069] At block 222, the motion image is changed to a still image. For example, if method 200 is performed when a motion image is captured by a client device, the motion image may be temporarily stored in a volatile memory (e.g., RAM) of the client device. If the motion image does not meet a storage threshold, only certain frames of the motion image are stored, e.g., written to a non-volatile memory of the client device. Other frames of the motion image are deleted before storing the image.
[0070] For example, the plurality of frames may include one or more frames taken before user activation of the client device's camera by the user, one frame taken at user activation, and one or more frames taken after user activation. For example, the length of the motion picture may be up to three seconds. In some implementations, the particular frames may be captured by the user It may be a frame captured at startup, e.g., associated with a timestamp that coincides with the startup of the camera. In some implementations, the particular frame may be any other frame of a plurality of frames of a video sequence.
[0071] For example, the particular frames may be selected based on different characteristics of the video imagery determined based on a programmatic analysis of the images. For example, such characteristics may include the amount of blur in the frame, whether the frame depicts a human subject with a smiling face and / or open eyes, how well the subject is focused, etc. Different characteristics of each frame may be evaluated to determine a quality score, and the frame associated with the highest quality score may be selected as the particular frame. In some implementations, the particular frames may be selected randomly.
[0072] Changing the motion image to a still image provides technical advantages. For example, because a still image contains one frame, the file size of a still image is smaller than a motion image containing multiple frames. Therefore, less storage space is required to store the still image. Furthermore, by transmitting the still image to, for example, a server device or other client device, network utilization is reduced. Block 222 may be followed by block 224.
[0073] In block 224, the image is stored, for example, in non-volatile memory of the client device 120, non-volatile memory of the server device, or both. Block 224 may be performed after block 220 to store a moving image (comprising multiple frames), and block 224 may be performed after block 222 to store a still image.
[0074] Although the method 200 has been described with reference to various blocks in Figure 2, it should be understood that the techniques described in this disclosure may be practiced without performing some of the blocks in Figure 2. In various implementations, some of the blocks of the method 200 may be performed in parallel or in an order different from that shown in Figure 2.
[0075] For example, in some implementations where method 200 is performed to display a user interface including one or more pre-stored images, blocks 220-224 may not be performed. In another example, blocks 220 and 222 are not performed and block 224 is performed after block 216 or block 218, for example, if method 200 is performed immediately after a motion image is captured and the user has indicated that the motion image is to be saved as it is captured. In some implementations, block 214 is not performed and block 212 is performed following block 216, for example, if the user has indicated that the motion image is to be displayed with subject motion at all times. In yet another variation of these implementations, block 216 may be performed to display the motion image and block 212 may be performed after block 216 to determine the motion score.
[0076] Further, some implementations may omit displaying the motion images entirely, e.g., in such implementations, blocks 214-218 are not performed. In these implementations, a motion score is determined for the motion images (block 212), and if the motion score meets a storage threshold (block 220), the motion images are selectively converted to still images (block 222). The motion or still images are stored in non-volatile memory (block 224). For example, a server device may utilize these implementations to conserve storage usage. In these implementations, the images may be displayed after they are stored, e.g., if the images were stored as motion images, they are displayed as motion images, and if the images were stored as still images, they are displayed as still images.
[0077] Furthermore, in some implementations, blocks 214-218 may be performed after blocks 212 and 220-224, for example, to display previously stored motion images. In these implementations, the motion scores may be stored with the images or may be recalculated upon display of the images.
[0078] FIG. 3 is a flow diagram illustrating an example of a method 300 for determining a motion score of a video according to some embodiments. In some embodiments, the method 300 may be implemented, for example, on the server system 102 shown in FIG. 1. In some embodiments, some or all of the method 300 may be implemented on one or more client devices 120, 122, 124, or 126 shown in FIG. 1, on one or more server devices, and / or on both the server device(s) and the client device(s). In the described example, the implementation system includes one or more digital processors or processing circuits ("processors") and one or more storage devices (e.g., database 106 or other storage). In some embodiments, different components of one or more servers and / or one or more clients may perform different blocks or other portions of the method 300. In some examples, a first device is described as performing blocks of the method 300. Some embodiments may have one or more blocks of the method 300 performed by one or more other devices (e.g., other client devices or server devices) that may send results or data to the first device.
[0079] In some implementations, method 300 or portions thereof may be automatically initiated by a system. In some implementations, the implementation system is the first device. For example, the method (or portions thereof) may be executed periodically or based on one or more specific events or conditions, such as, for example, an application (e.g., camera application 152, image management application 156, etc., being initiated by a user), the receipt of one or more images newly uploaded to or accessible by the system, a predetermined amount of time having elapsed since method 300 was last executed, and / or the occurrence of one or more other conditions that may be specified in the settings that the method loads. In some implementations, such conditions may be specified by a user in the user's stored custom preferences.
[0080] In various implementations, client device 120 may be a standalone camera, a mobile phone, a smartphone, a tablet computer, a wearable device, or other client device capable of receiving content input by a user to the client device (e.g., taking an image) and performing method 300. In another example, a client device or a server device may perform method 300. Some implementations may initiate method 300 based on user input. A user (e.g., an operator or end user) may, for example, select to initiate method 300 from a display user interface, such as an application user interface or other user interface. In some implementations, method 300 may be implemented by a client device. In some implementations, method 300 may be implemented by a server device.
[0081] At block 302, a degree of stability of the video image is determined. In some implementations, the degree of stability may be based on a comparison of pixel values of corresponding pixels in consecutive frames of the video image. For example, the image may be determined to be stable if a significant percentage (e.g., 90%) of the pixels match, e.g., have equal or similar values. In some implementations, the video image may be programmatically analyzed to determine background portions of the video image. In these implementations, the comparison of pixel values may be performed for only a subset of pixels that correspond to background portions.
[0082] In some implementations, the degree of stability may be determined based on data obtained from one or more sensors of the device capturing the video images, e.g., client device 120. For example, such data may include accelerometer and / or gyroscope readings of the device. For example, a high degree of stability may be determined if data from the sensors indicates that the device is not moving or rotating during capture of the video images (which may occur for, e.g., a short period of time, e.g., up to 3 seconds). On the other hand, movement of the device during capture (e.g., as indicated by device shifts and / or rotations measured by the sensors) may indicate a low degree of stability.
[0083] In some implementations, the determination of the degree of stability may be based on pixel values and sensor readings of multiple frames. Other techniques for determining the degree of stability may also be used. Block 302 may be followed by block 304.
[0084] In block 304, it may be determined whether the video has stability. For example, the degree of stability of the video may be compared to a stability threshold. If the degree of stability meets the stability threshold, block 304 is followed by block 306. Otherwise, block 304 is followed by block 330.
[0085] At block 306, the video is programmatically analyzed to detect video objects and determine a type of object. For example, object recognition techniques may be used to detect the presence of various objects in the video, such as human faces, human bodies, animals, trees, and inanimate objects, such as buildings, furniture, vehicles, etc. The video object may be determined as one or more objects that appear near the center of multiple frames, the object that is in focus, the object that occupies the most area in the image, etc. For example, video depicting a human face that is in focus and / or in the center may correspond to a subject type of "human face" or "portrait." In another example, video depicting a group of people may correspond to a subject type of "group photo." In yet another example, video depicting dogs, cats, or other animals may correspond to a subject type of "pets" or "animals," etc.
[0086] Also, multiple frames of a video may be analyzed to determine the degree of subject movement within the video. For example, if the type of subject is a "human face," the degree of movement may be determined based on changes in pixel values of the face in multiple frames. In this example, video with little or no change in facial expression may correspond to low subject movement, and video in which the subject smiles, raises eyebrows, turns or tilts their face, etc. may correspond to high subject movement.
[0087] In another example, if the object type is a human body, then object movements such as jumping or moving from one location to another may correspond to a high degree of movement, while stationary objects that move in other ways (e.g., raising a hand, tilting a head, etc.) may correspond to a low degree of movement. In some implementations, the threshold level of movement used for the comparison may be based on the object type, e.g., a lower threshold may be used if the object is a face than if it is a body. Various techniques may be used to detect object movement, e.g., optical flow, image segmentation, image registration, etc. For example, object recognition techniques may be used to detect the optical flow of an object and pixels corresponding to the object, e.g., measuring the outline of the object to determine the degree of object movement. In another example, image registration techniques may be used to align pixels of multiple frames, and the change in position of the object indicated by different pixel coordinates may be used to determine the degree of movement. In some implementations, a combination of techniques may be used. Block 306 may be followed by block 308.
[0088] Block 308 determines whether the degree of object movement meets a threshold level of movement. If the degree of object movement meets the threshold, block 308 is followed by block 310. Otherwise, block 308 is followed by block 330.
[0089] In block 310, it is determined whether the object type is landscape or inanimate. For example, if the detected objects include objects such as mountains, trees, rivers, and oceans, the object type of the video image is determined to be landscape, for example, and inanimate if the detected objects include objects such as furniture, buildings, and automobiles, for example. In some implementations, for example, if a person or animal (e.g., a pet) is detected in an image having landscape or inanimate objects, the object type is not landscape or inanimate because a person or animal object is detected. If the object type is landscape or inanimate, block 310 is followed by block 330. Otherwise, block 310 is followed by block 312.
[0090] Block 312 determines whether the number of views of the video meets the audience threshold. For example, the video is determined to meet the audience threshold if multiple frames of the video have previously been viewed consecutively, thereby providing an indication of subject movement at least a threshold number of times. For example, a stored image that has been viewed multiple times by a user (or other users who have shared the image), e.g., five times, ten times, etc., may be determined to meet the audience threshold, such that movement in the image is of interest to the user. If the number of views of the video meets the audience threshold, block 312 may be followed by block 320. Otherwise, block 312 may be followed by block 314.
[0091] At block 314, it is determined whether a motion artifact was created from the video. For example, the motion artifact may include an animated GIF or other animation, for example, including two or more frames of the video. For example, the motion artifact may include two or more frames set to play in a loop. The two or more frames of the loop may be played in sequence, for example, after a first frame, a second frame may be played, followed by a third frame. The motion artifact may be a forward loop, where a first frame is provided after a last frame, to provide a continuous non-stop animation. The motion artifact may be a reverse loop (also known as a boomerang), where frames are played in sequence from a first frame to a last frame, followed by a reverse sequence from the last frame to the first frame, to provide a continuous animation. In some implementations, the motion artifact may include other creations based on the video, such as, for example, a collage including two or more frames of the video, a motion blur image where two or more frames of the video are superimposed into a single still image, etc.
[0092] A camera application, image gallery application, or other application may provide functionality for creating a motion artifact, for example, automatically or manually. If a motion artifact has been created, for example, if a user has selected to create the motion artifact by providing input or to save an automatically generated motion artifact, block 314 may be followed by block 316. Otherwise, block 314 may be followed by block 320.
[0093] At block 316, a motion score is calculated for the video. In some implementations, the motion score may be based on one or more of a degree of stability of the video, a type of object in the video, or a degree of object movement in multiple frames of the video. The degree of stability may be a weighted combination of the degree of stability and the degree of object movement in the motion. In some implementations, a greater degree of stability may correspond to a higher motion score and a lesser degree of stability may correspond to a lower motion score. In some implementations, a greater degree of object movement may correspond to a higher motion score and a lesser degree of object movement may correspond to a lower motion score. In some implementations, different weights may be assigned to the degree of object movement based on the type of object. For example, if the object type is a face, a greater weight may be assigned to the degree of movement than if the object type is a body.
[0094] In some implementations, one or more other factors may be utilized in addition to the factors described above. For example, if the user permits the use of face recognition technology, it may determine whether the subject of the image is a person known to the user (e.g., part of the user's social network as depicted in other images taken by the user or stored in the user's image gallery, etc.) or an animal (e.g., a pet). Video images depicting known subjects may have a higher motion score than video images depicting unknown subjects, other factors being equal. In some implementations, video images depicting certain types of subjects, e.g., sports or exercise, dance, etc., may be associated with a higher weight, and video images depicting such types of subjects may receive a correspondingly higher motion score. In other words, a higher motion score indicates a greater or more substantial frame-to-frame variation in the video image as perceived by the user, and when displayed, the motion aspects of such images increase or enhance the user experience, perception and information gained by the user from the motion aspects. On the other hand, a low motion score indicates little or insignificant variation between frames of a moving image, and the loss of the motion information, either by static display or removal and conversion of the moving image to a still image, will not degrade or even improve the user experience and perception and recognition of the image content. In the latter case, for example, the motion aspects contained in the moving image may include a jitter caused by the jitter or shudder of the camera when the moving image of the static scene was taken. In this situation, for example, the motion information and the display of the image as a moving image will actually degrade the user's perception, since the jitter caused by the motion aspects will prevent the user from properly concentrating and recognizing the details of the static scene.
[0095] In some implementations, the video images may include or be associated with a label indicating the type of object (e.g., a label "archive" indicating the type of object is a document, screenshot, receipt, etc.), and such a label may be used to determine the motion score. In some implementations, the video images may include sounds, for example, recorded during the capture of the video images. In these implementations, the sounds may be programmatically analyzed to determine whether they correspond to a category of sounds associated with motion of interest, e.g., sounds of a waterfall, a bat hitting a ball, dance music, laughter, etc. may be associated with motion of interest, while other sounds, such as background noise, vehicle noise, etc., may be associated with images that do not depict motion of interest.
[0096] In some implementations, the calculated motion scores may be stored, e.g., in association with images included in the video metadata, as labels associated with the video, in a database that stores data about the video, etc. The stored motion scores may be accessed, e.g., when generating a user interface that includes the video, when suggesting the creation of motion artifacts or other image creations, etc.
[0097] In block 320, which is performed if the number of views of the video meets the viewer count threshold or if motion artifacts are created from the video, the motion score may be set to a value that meets the display threshold. The motion score may be calculated as described above and compared to a display threshold. If the motion score does not meet the display threshold, the motion score may be set to a value that meets the display threshold. Updating the motion score in this manner ensures that user input indicating interest in the video takes precedence over an automatically determined motion score, and by displaying multiple frames of the video in succession, video of interest to the user is displayed in a manner that the user perceives as motion.
[0098] Block 330 is executed if the motion image is not stable (as determined in block 304), if the degree of object movement does not meet a threshold level of movement (block 308), or if the type of object in the motion image is landscape or inanimate (block 310). In block 330, the motion score may be set to a value that does not meet the display threshold, e.g., a value lower than the display threshold. Setting the motion score in this way can ensure that motion images that do not allow a high-quality viewing experience by displaying only certain frames of the motion image, e.g., images that are unstable or not stable, have minimal object movement, or have object types that are not suitable for displaying multiple frames of the motion image, are displayed as still images, e.g., as described above with reference to FIG. 2. Setting the motion score in this way can save computational resources for rendering multiple frames of images, or can save storage space when the motion image is stored in non-volatile memory or transmitted to a server or other client device.
[0099] Although the method 300 has been described with reference to various blocks in FIG. 3, it should be understood that the techniques described in this disclosure may be practiced without performing some of the blocks in FIG. 3. For example, in some implementations, blocks 312, 314, and 320 are not performed, e.g., if the number of views or information related to the creation of motion artifacts is not available. In some implementations, one or more of the blocks shown in FIG. 3 may be combined. In various implementations, some of the blocks of the method 300 may be performed in parallel or in a different order than that shown in FIG. 3.
[0100] FIG. 4 is a flow diagram illustrating an example method 400 for determining a motion score of a video according to some embodiments. In some embodiments, the method 400 may be implemented, for example, on the server system 102 shown in FIG. 1. In some embodiments, some or all of the method 400 may be implemented on one or more client devices 120, 122, 124, or 126 shown in FIG. 1, on one or more server devices, and / or on both the server device(s) and the client device(s). In the described example, the implementation system includes one or more digital processors or processing circuits ("processors") and one or more storage devices (e.g., database 106 or other storage). In some embodiments, different components of one or more servers and / or one or more clients may perform different blocks or other portions of the method 400. In some examples, a first device is described as performing blocks of the method 400. Some embodiments may have one or more blocks of the method 400 performed by one or more other devices (e.g., other client devices or server devices) that may send results or data to the first device.
[0101] In some implementations, method 400 or portions thereof may be automatically initiated by a system. In some implementations, the implementation system is the first device. For example, the method (or portions thereof) may be executed periodically, or in response to, for example, an application (e.g., camera application 152, image management application 156, etc., being initiated by a user), one or more images newly uploaded to or accessible by the system, a predetermined amount of time having elapsed since method 400 was last executed, and / or one or more other events that may be specified in the settings that the method loads. or other condition has occurred. In some implementations, such conditions may be specified by the user in the user's stored custom preferences.
[0102] In various implementations, client device 120 may be a standalone camera, a mobile phone, a smartphone, a tablet computer, a wearable device, or other client device capable of receiving content input by a user to the client device (e.g., taking an image) and performing method 400. In another example, a client device or a server device may perform method 400. Some implementations may initiate method 400 based on user input. A user (e.g., an operator or end user) may, for example, select to initiate method 400 from a display user interface, such as an application user interface or other user interface. In some implementations, method 400 may be implemented by a client device. In some implementations, method 400 may be implemented by a server device.
[0103] Block 402 determines whether the number of views of the video meets the audience threshold. Block 402 may be similar to block 312 described in Figure 3. If the number of views meets the audience threshold, block 402 is followed by block 410. Otherwise, block 402 is followed by block 404.
[0104] Block 404 determines whether motion artifacts have been created from the video image. Block 404 may be similar to block 314 described in Figure 3. If at least one motion artifact has been created, block 404 is followed by block 410. Otherwise, block 404 is followed by block 410.
[0105] Block 406 determines whether a user input has been received to display the motion image as a still image, for example, by displaying a particular frame of the motion image. For example, a user may provide such input by selecting an option such as "turn off motion in this image." Such user input may be an indication that a viewer will not notice motion in the image of interest. If user input has been received, block 406 is followed by block 420. Otherwise, block 406 may be followed by block 408.
[0106] At block 408, a motion score is calculated for the video image. In some implementations, the motion score is calculated by applying a trained machine learning model that is comprised of a neural network that includes multiple nodes organized into multiple layers and is configured to determine a motion score for the image. The video image is provided as an input to the trained machine learning model.
[0107] Some implementations include training a machine learning model to determine a motion score for a video. For example, the untrained machine learning model may be comprised of a neural network including a plurality of nodes organized into a plurality of layers. The nodes of each layer may be connected to the nodes of the previous layer and the nodes of the subsequent layer. The nodes of the first layer may be configured to accept the video as an input, and the nodes of the last layer of the plurality of layers may be configured to output a motion score. Each node may be any type of neural network node, for example, an LSTM node.
[0108] Prior to training, an initial weight may be assigned to each of the nodes and connections between the nodes in different layers of the neural network may be initialized. A training dataset of video images may be obtained and The training dataset may include a plurality of video images and associated motion scores or labels. For example, a first set of video images in the training dataset may be associated with one or more labels that indicate that the motion of the object in each of the images is of interest to the user. Such labels may be obtained based on user input or may be obtained from user data obtained with user permission (e.g., number of viewers of the video images, creation of motion artifacts, etc.).
[0109] The second set of video images of the training data set may be associated with one or more labels indicating that the motion of the object in each of the second set of images is not of interest to the user. These labels may be obtained based on user input or may be obtained from user data obtained with the user's permission (e.g., discarding frames of the video, deleting the video, etc.). The first set may be referred to as "active training examples" and the second set may be referred to as "passive training examples." The training set may include video images depicting different types of objects with different degrees of stability and degree of object movement within the multiple frames. The training set may be selected to ensure that the training set includes examples of different types of images that may later be provided as inputs to the model during field use.
[0110] A number of video images may be provided as input to the model during training. In some implementations, training images and associated labels may first be provided as input to the model during training. During this training phase, the weights of individual nodes (and optionally the number of nodes), the connectivity between nodes in different layers (or optionally the number of layers) may be automatically adjusted so that the motion scores output by the model match the motion scores contained in the associated labels. In some implementations, this may be referred to as the first training phase.
[0111] In some implementations, a subset of the training set of images may be excluded from the initial training stage. This subset may be provided after the initial training stage, and the motion scores output by the model may be compared with the motion scores associated with the images included in the subset. If the motion scores do not match, such mismatches may be provided as inputs for training to the model (reinforcement learning). In this stage, model parameters may be adjusted, such as weights of one or more nodes, connectivity between nodes, etc., until the model is able to correctly predict the motion scores of the subset of images. For example, the second stage may be repeated any number of times until the model achieves a sufficient level of accuracy in predicting the motion scores. In some implementations, the trained model may be further modified, e.g., compressed (to use fewer nodes or layers), transformed (e.g., to be usable on different types of hardware), etc. The trained model may be provided on a device, e.g., the client device 120 or a server device, to calculate the motion scores of the video images. In some implementations, different versions of the model may be provided. For example, the client version of the model may be optimized for size to reduce computational complexity, while the server version of the model may be optimized for accuracy.
[0112] At block 408, the video is provided as input to a trained machine learning (ML) model. The trained ML model may calculate a motion score based on the video. In some implementations, instead of or in addition to the video, other data about the video may be provided as input to the trained ML model. For example, such factors may include a degree of stability (e.g., of background portions) of the video, a type of object in the video, or a degree of object movement in the video. In different implementations, such factors may be determined based on pixel values of the image using the techniques described with reference to FIG. 3. In some implementations, separate ML models trained to determine the degree of stability, type of object, or degree of object movement may be provided, and the output of these ML models may be calculated based on the motion score. The forces may be provided to a trained ML model, which in some implementations may determine such factors as outputs of intermediate layers.
[0113] If the number of views of the video meets the viewer count threshold (block 402) or if motion artifacts are created from the video (block 404), block 410 is executed. Block 410 sets a motion score to a value that meets the display threshold, similar to block 320 described in Figure 3. Block 410 may be followed by block 430.
[0114] Block 420 may be followed by block 430, for example by setting the motion score to a value that does not meet the display threshold, similar to block 330 described in Figure 3.
[0115] In block 430, the trained ML model may be updated. For example, when block 430 is executed after block 410, the model may be updated based on the motion score being set to a value that meets a display threshold (based on user input), thereby indicating that the video depicts a moving image of interest to the user. In another example, when block 420 is executed after block 420, the model may be updated based on the motion score being set to a value that does not meet a display threshold, thereby indicating that the video depicts a moving image of interest to the user.
[0116] In some implementations, updating the ML model may include adjusting a weight of at least one of the plurality of nodes or updating a connection between at least one pair of the plurality of nodes. In some implementations, the weight may be adjusted such that the ML model calculates a motion score after updating that matches the motion score set in block 410 or 420. In some implementations, block 430 is not performed, for example, if the user does not provide permission to use the user data. In implementations in which the user allows the use of user data, the updated ML model may be provided to a server, for example, to enable federated learning. Updating the trained ML model in this way may enable personalized calculation of the motion score for a user viewing the video, thereby enabling improved video display with reduced computational load or reduced storage costs, with the motion score determined by the ML model accurately indicating the user's level of interest in the video captured or viewed.
[0117] Although method 400 has been described with reference to various blocks in FIG. 4, it should be understood that the techniques described in this disclosure may be practiced without performing some of the blocks in FIG. 4. For example, in some implementations, blocks 402 and / or 404 are not performed. In some implementations, one or more of the blocks shown in FIG. 4 may be combined. In various implementations, some of the blocks of method 400 may be performed in parallel or in a different order than the order shown in FIG. 4. For example, in some implementations, blocks 402 and 404 may be performed simultaneously.
[0118] FIG. 5 illustrates an exemplary video sequence. In particular, three frames (502, 512, and 522) of the exemplary video sequence are shown. The three frames may be captured at different times within a short period of time, e.g., 1-3 seconds. As seen in FIG. 5, the subject of the video sequence (506, 516, 526) is a person on a swing. During capture of the video sequence, the person swings from back to front and spreads their legs apart, as seen in frames 502, 512, and 522. This movement may be indicated by depth data stored in a depth image, for example, if the video sequence was captured using a camera capable of determining depth information. The video sequence also includes a background portion (504).
[0119] The video footage shown in Figure 5 includes objects (people) with significant motion and may be determined to have a relatively high motion score that meets, for example, a display threshold. Other video footage may include objects, e.g., documents, furniture, or other inanimate objects, or scenery, or may not include significant object motion and may be determined to have a relatively low motion score that does not meet, for example, a display threshold. While Figure 5 shows a stable image (wherein the background 504 does not change between frames 502, 512, and 522), other video footage may include unstable or non-stable images, for example, where the background changes between frames.
[0120] Figure 6 is an illustration of an example user interface 600 that includes an image grid. The example user interface shown in Figure 6 includes the motion image shown in Figure 5 as part of the image grid. Three views (602, 612, 622) of the user interface are shown to illustrate that the user interface displays multiple frames of the motion image in succession. In this example, the motion image is displayed at a larger size than the other images included in the image grid.
[0121] In this exemplary user interface, one motion image is shown with subject motion displayed in the user interface. However, it should be understood that any number of images with subject motion may be shown, e.g., zero, two, three, or more images, based on the motion scores of each of the images in the image grid, as described with reference to FIG. 3. In various implementations, the user interface 600 may be displayed by a display device, e.g., a display screen of the client devices 120, 122, 124, and / or 126 or the server system 102 of FIG. 1.
[0122] FIG. 7 is a block diagram of an exemplary device 700. The device 700 may be used to implement one or more features described herein. In one example, the device 700 may be used to implement a client device, such as any of the client devices (120, 122, 124, 126) shown in FIG. 1. Alternatively, the device 700 may implement a server device, such as the server 104. In some implementations, the device 700 may be used to implement a client device, a server device, or both a client and a server device. The device 700 may be any suitable computer system, server, or other electronic device or hardware device described above.
[0123] One or more methods described herein may be a standalone program that can run on any type of computing device, a program running on a web browser, a mobile application ("app") running on a mobile computing device (e.g., a mobile phone, a smart phone, a tablet computer, a wearable device (e.g., a watch, an arm band, jewelry, a hat, virtual reality goggles or glasses, augmented reality goggles or glasses, a head mounted display, etc.), a laptop computer, etc.). In one example, a client / server architecture may be used, e.g., a mobile computing device (as a client device) sends user input data to a server device and receives final output data for output (e.g., for display) from the server. In another example, all computations may be performed within the mobile app (and / or other apps) on the mobile computing device. In another example, computations may be shared between the mobile computing device and one or more server devices.
[0124] In some implementations, the device 700 includes a processor 702, a memory 704, and The processor 702 includes an input / output (I / O) interface 706 and a camera 716. The processor 702 may be one or more processors and / or one or more processing circuits for executing program code and controlling basic operations of the device 600. A "processor" includes any suitable hardware system, mechanism, or component for processing data, signals, or other information. The processor may include a system with a general-purpose central processing unit (CPU) having one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit for realizing a function, a specialized processor for performing neural network model-based processing, a neuron circuit, a processor optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some implementations, the processor 702 may include one or more co-processors for implementing neural network processing. In some implementations, the processor 702 may be a processor that processes data to generate a probability-based output, e.g., the output generated by the processor 702 may be imprecise or may be accurate within a particular range from an expected output. The processing need not be limited to a particular geographic location or be time-bound. For example, the processor may perform its functions in "real-time", "offline", "batch mode", etc. Portions of the processing may be performed at different times and in different locations by different (or the same) processing systems. A computer may be any processor in communication with a memory.
[0125] Memory 704 is typically provided within device 700 for access by processor 702. Memory 704 may be any suitable processor-readable storage medium suitable for storing instructions executed by a processor, such as Random Access Memory (RAM), Read-Only Memory (ROM), Electrical Erasable Read-only Memory (EEPROM), flash memory, etc., and may be located separately from and / or integrated with processor 702. Memory 704 may store software operated on server device 700 by processor 702, including an operating system 708, machine learning applications 730, other applications 712, and application data 714. Other applications 712 may include applications such as a camera application, an image gallery or library application, a data display engine, a web hosting engine, an image display engine, a notification engine, a social networking engine, etc. In some implementations, the machine learning application 730 and the other applications 712 may each include instructions that enable the processor 702 to perform the functions described herein, e.g., some or all of the methods of FIGS. 2, 3, and 4.
[0126] Other applications 712 may include, for example, a camera application, an image library or gallery application, a media viewing application, a communication application, a web hosting engine or application, a mapping application, a media sharing application, etc. One or more methods disclosed herein can operate in a number of environments and platforms, such as, for example, as a standalone computer program that can operate on any type of computing device, as a web application having a web page, as a mobile application ("app") that operates on a mobile computing device, etc.
[0127] In various implementations, the machine learning application may utilize a Bayesian classifier, a support vector machine, a neural network, or other learning techniques. In some implementations, the machine learning application 730 may include a trained model 734, an inference engine 736, and data 732. In some implementations, the data 732 may include training data, e.g., data used to generate the trained model 734. For example, the training data may include any type of data, such as text, images, audio, video, etc. For example, the training data may include video images. The training data may be obtained from any source, e.g., a data repository specifically marked for training, data that has been given permission to be used as training data for machine learning, etc. In implementations where one or more users have authorized the use of their user data to train a machine learning model, e.g., the trained model 734, the training data may include user data. In embodiments in which users have authorized the use of their respective user data, data 732 may include authorization data, such as images or image metadata (e.g., video, motion artifacts, or other user generated images, data related to sharing the images with other users, labels associated with the images, etc.), communications (e.g., emails; chat data such as text messages, audio, video, etc.), documents (e.g., spreadsheets, text documents, presentations, etc.), etc.
[0128] In some implementations, the training data may include synthetic data generated for training purposes, such as data that is not based on user input or activity in the context to be studied, e.g., data generated from simulated video or computer-generated video. In some implementations, the machine learning application 730 excludes the data 732. For example, in these implementations, the trained model 734 may be generated, for example, on a different device and provided as part of the machine learning application 730. In various implementations, the trained model 734 may be provided as a data file that includes a model structure or model form (e.g., specifying the number and type of neural network nodes, the connectivity between the nodes, and the organization of the nodes into multiple layers) and associated weights. The inference engine 736 may read the data file for the trained model 734 and implement a neural network with node connectivity, layers, and weights based on the model structure or model form specified in the trained model 734.
[0129] The machine learning application 730 may also include a trained model 734. In some implementations, the trained model may include one or more model forms or one or more model structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep neural network implementing multiple layers (e.g., a "hidden layer" between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (e.g., a network that splits or partitions input data into multiple portions or tiles, processes each tile separately using one or more neural network layers, and aggregates the results from the processing of each tile), a sequence-to-sequence neural network (e.g., a network that takes sequential data as input, such as words in a sentence or frames in a video, and produces a result sequence as output), etc. The model form or structure may specify the connectivity between various nodes and the organization of the nodes into layers. For example, nodes in a first layer (e.g., an input layer) may receive data as input data 732 or application data 714. Such data may include, for example, one or more pixels per node, for example when the trained model is used for image analysis of video images containing multiple frames. Subsequent intermediate layers may receive as input the outputs of the nodes of the previous layer according to the connectivity specified in the model form or structure. These layers may also be classified as hidden layers. The final layer (e.g., the output layer) generates the output of the machine learning application. Depending on the particular trained model, this output may be, for example, a motion score, a set of labels for the image, a display of the image that allows comparison of the image with other images (e.g., the image's feature vectors), an output sentence responsive to an input sentence, one or more categories of input data, etc. In some implementations, the model topology or structure also specifies the number and / or type of nodes in each layer.
[0130] In different implementations, the trained model 734 may include multiple nodes arranged in multiple layers according to a model structure or model morphology. In some implementations, the node may be a memoryless computational node configured to process, for example, a unit of input and generate a unit of output. The computation performed by the node may include, for example, multiplying each of the multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or intercept value to generate the node output. In some implementations, the computation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some implementations, the step / activation function may be a non-linear function. In various implementations, such computations may include operations such as matrix multiplication. In some implementations, the computations by the multiple nodes may be performed in parallel, for example, using multiple processor cores in a multi-core processor, or using individual processing units of a GPU or specialized neuron circuit. In some implementations, the node may include memory, for example, to store one or more previous inputs for use in processing subsequent inputs. For example, the node with memory may include a Long Short-Term Memory (LSTM) node. The LSTM node uses memory to train the node in a Finite State Machine (FSM). A model may maintain "state" that allows it to operate like a state machine. A model with such nodes may be useful for processing sequential data, such as, for example, multiple words in a sentence or paragraph, multiple frames in a video, or speech or other audio.
[0131] In some implementations, the trained model 734 may include embeddings or weights for individual nodes. For example, the model may be initialized as a number of nodes organized into layers as specified by the model topology or model structure. At initialization, weights may be applied to connections between pairs of nodes connected according to the model topology, e.g., pairs of nodes in successive layers of a neural network. For example, each weight may be randomly assigned or initialized to a default value. The model may then be trained, e.g., with data 732, to generate results.
[0132] For example, the training step may include applying a supervised learning technique. In supervised learning, the training data may include multiple inputs (e.g., a set of images) and corresponding expected outputs for each input (e.g., one or more labels for each image). For example, the values of the weights are automatically adjusted based on comparing the model's outputs to expected outputs to increase the probability that the model will generate the expected outputs given similar inputs.
[0133] In some implementations, the training step may include applying an unsupervised learning technique, where only the input data is given and the model may be trained to distinguish between the data, e.g., cluster the input data into groups, each group containing input data that is similar in some way. For example, the model may be trained to assign scores to video images.
[0134] In various embodiments, the trained model includes a set of weights or a set of embeddings that correspond to the model structure. In embodiments where data 732 is excluded, the machine learning application The application 730 may include a pre-trained model 734 that is based on prior training, e.g., by the developer of the machine learning application 730, a third party, etc. In some implementations, the pre-trained model 734 may include a fixed set of weights, e.g., downloaded from a server that provides the weights.
[0135] The machine learning application 730 also includes an inference engine 736. The inference engine 736 is configured to apply the trained model 734 to data, such as the application data 714 (e.g., video images), to provide inferences. In some implementations, the inference engine 736 may include software code executed by the processor 702. In some implementations, the inference engine 736 may specify circuitry (e.g., a programmable processor, a field programmable gate array (FPGA), etc.) that enables the processor 702 to apply the trained model. In some implementations, the inference engine 736 may include software instructions, hardware instructions, or a combination. In some implementations, the inference engine 736 may provide an application programming interface (API) that the operating system 708 and / or other applications 712 can use to call the inference engine 736, for example, to apply the trained model 734 to the application data 714 to generate inferences.
[0136] The machine learning application 730 may provide several technical advantages. For example, if the trained model 734 is generated based on unsupervised learning, the trained model 734 may be applied by the inference engine 736 to generate a knowledge representation (e.g., a numerical representation) from input data, e.g., application data 714. For example, a trained model for image analysis may generate a representation of an image that is smaller in data size (e.g., 1 KB) than the input image (e.g., 10 MB). In some implementations, such a representation may help reduce the processing cost (e.g., computational cost, memory usage, etc.) of generating an output (e.g., image label, classification, sentence description, etc.). In some implementations, such a representation may be provided as input to a different machine learning application that generates an output from the output of the inference engine 736. In some implementations, the knowledge representation generated by the machine learning application 730 may be provided to a different device that performs further processing, e.g., over a network. In such implementations, providing a knowledge representation rather than an image may provide technical advantages, e.g., allowing faster data transmission at less cost. In another example, a document clustering trained model may generate document clusterings from input documents that are likely suitable for further processing without needing access to the original documents (e.g., determining whether a document is topical, determining the taxonomic category of the document, etc.), thus saving computational costs.
[0137] In some implementations, the machine learning application 730 may be implemented offline. In these implementations, the trained model 734 may be generated in a first stage and provided as part of the machine learning application 730. In some implementations, the machine learning application 730 may be implemented online. For example, in such implementations, an application that invokes the machine learning application 730 (e.g., the operating system 708, one or more of the other applications 712) may use the inferences generated by the machine learning application 730, e.g., provide the inferences to a user, and generate a system log (e.g., actions taken by the user based on the inferences, if permitted by the user; or results of further processing, if used as input for such further processing). The system log may be generated periodically, e.g., hourly, monthly, quarterly, etc., and may be generated periodically, e.g., hourly, monthly, quarterly, etc., and may be generated at any time, depending on the user's permission. , may be used to update the trained model 734, e.g., to update the embeddings for the trained model 734.
[0138] In some implementations, the machine learning application 730 may be implemented to adapt to the particular configuration of the device 700 on which it executes. For example, the machine learning application 730 may determine a computation graph that utilizes available computational resources, e.g., the processor 702. For example, if the machine learning application 730 is implemented as a distributed application on multiple devices, the machine learning application 730 may determine that the computations are performed in a manner that optimizes the computations on each individual device. In another example, the machine learning application 730 may determine that the processor 702 includes a GPU with a particular number of GPU cores (e.g., 1000) and implement the inference engine accordingly (e.g., as 1000 individual processes or threads).
[0139] In some implementations, the machine learning application 730 may implement a set of trained models. For example, the trained models 734 may include multiple trained models, each applicable to the same input data. In these implementations, the machine learning application 730 may choose a particular trained model based on, for example, available computational resources, success rate with previous inferences, etc. In some implementations, the machine learning application 730 may execute an inference engine 736 such that multiple trained models are applied. In these implementations, the machine learning application 730 may combine the outputs from applying the individual models, for example, using a majority vote to score the outputs from applying each trained model, or by choosing one or more particular outputs. Furthermore, in these implementations, the machine learning application may apply a time threshold (e.g., 0.5 milliseconds) for applying the individual trained models and utilize only individual outputs that are available within the time threshold. Outputs not received within the time threshold may not be used, e.g., may be discarded. For example, such an approach may be appropriate when there is a time limit specified during invocation of the machine learning application, for example, by the operating system 708 or one or more applications 712.
[0140] In different implementations, the machine learning application 730 can generate different types of output. For example, the machine learning application 730 can provide a representation or cluster (e.g., a numerical representation of the input data), a label (e.g., of the input data including images, documents, etc.), a phrase or sentence (e.g., a description of an image or video suitable for use as a response to an input sentence), an image, audio, or video (e.g., generated by the machine learning application in response to the input) (e.g., in response to an input video, the machine learning application 730 may generate an output video with a particular effect applied, e.g., an output video drawn in the style of a comic book or a particular artist, such as when the trained model 734 is trained with training data from a comic book or a particular artist). In some implementations, the machine learning application 730 may generate the output based on a format specified by an invoking application, e.g., the operating system 708 or one or more applications 712. In some implementations, the invoking application may be another machine learning application. For example, such a configuration may be used in a generative adversarial network in which an invoking machine learning application is trained using output from machine learning application 730, or in which machine learning application 730 is trained using output from an invoking machine learning application.
[0141] The software in memory 704 may alternatively be stored in any other suitable storage location or on any computer. The information stored in the memory 704 may be stored on a data-readable medium. Additionally, the memory 704 (and / or other connected storage device(s)) may store one or more messages, one or more taxonomies, an electronic encyclopedia, a dictionary, a thesaurus, a knowledge base, message data, grammars, user preferences, and / or other instructions and data used in the features described herein. The memory 704 and other types of storage, such as magnetic disks, optical disks, magnetic tapes, or other tangible media, may be considered "storage" or "storage devices."
[0142] The I / O interface 706 can provide functionality that allows the server device 700 to interface with other systems and devices. The interfaced devices can be included as part of the device 700. Or, the interfaced devices can be separate and in communication with the device 700. For example, network communication devices, storage devices (e.g., memory and / or database 106), and input / output devices can communicate through the I / O interface 706. In some implementations, the I / O interface can connect to interface devices, such as input devices (keyboards, pointing devices, touch screens, microphones, cameras, scanners, sensors, etc.) and / or output devices (display devices, speaker devices, printers, motors, etc.).
[0143] Some examples of interfaced devices that may be connected to the I / O interface 706 include one or more display devices 720 that may be used to display content, such as images, videos, and / or user interfaces of output applications described herein. The display devices 720 may be connected to the device 700 via a local connection (e.g., a display bus) and / or via a networked connection and may be any suitable display device. The display devices 720 may include any suitable display device, such as an LCD, LED, or plasma display screen, a CRT, television, monitor, touch screen, 3D display screen, or other display device. For example, the display devices 720 may be a flat display screen provided on a mobile device, multiple display screens included in a goggle or headset device, or a monitor screen of a computing device.
[0144] The I / O interface 706 may interface to other input and output devices. Some examples include one or more cameras capable of capturing images. Some implementations may provide a microphone for recording sound (e.g., as part of a captured image, as a voice command, etc.), an audio speaker device for outputting sound, or other input and output devices.
[0145] Camera 716 may be any type of camera capable of capturing video images including multiple frames. In some implementations, camera 716 may include multiple lenses with different capabilities, such as front-facing versus rear-facing lenses, different zoom levels, image resolutions for captured images, etc. In some implementations, device 700 may include one or more sensors, such as a depth sensor, an accelerometer, a position sensor (e.g., GPS), a gyroscope, etc. In some implementations, one or more of these sensors may be operated in conjunction with the camera to obtain sensor measurements corresponding to different frames of a video image captured using the camera.
[0146] For ease of explanation, Figure 7 illustrates one block for each of the processor 702, memory 704, I / O interface 706, camera 716, and software blocks 708, 712, and 730. These blocks may include one or more may represent a processor or processing circuitry, one or more operating systems, one or more memories, one or more I / O interfaces, one or more applications, and / or one or more software modules. In other implementations, device 700 may not have all of the illustrated components and / or may have other elements, including other types of elements instead of or in addition to the elements shown herein. Although some components have been described as performing the blocks and operations described in some implementations herein, any suitable component or combination of components of environment 100, device 700, a similar system, or any suitable processor or processors associated with such a system may perform the described blocks and operations.
[0147] The methods described herein can be implemented by computer program instructions or code that can be executed on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuits) and can be stored on a computer program product that includes a non-transitory computer-readable medium (e.g., storage medium), such as a magnetic, optical, electromagnetic, or semiconductor storage medium, including semiconductor or solid-state memory, magnetic tape, removable computer disk, Random Access Memory (RAM), Read-Only Memory (ROM), flash memory, rigid magnetic disks, optical disks, solid-state memory drives, and the like. Also, the program instructions can be included or provided as electrical signals, for example, in a Software as a Service (SaaS) format delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more of the methods can be implemented in hardware (e.g., logic gates) or a combination of hardware and software. Exemplary hardware may include programmable processors (e.g., Field-Programmable Gate Arrays (FPGAs), Complex Programmable Logic Devices (CPLDs)), general purpose processors, graphics processors, Application Specific Integrated Circuits (ASICs), etc. One or more of the methods may be implemented as a component or part of an application running on the system, or may be implemented as an application or software operating in conjunction with other applications and the operating system.
[0148] Although a description of specific implementations has been provided, these specific implementations are exemplary only and not limiting, and the illustrated concepts may be applied to other examples and implementations.
[0149] In situations where certain implementations described herein may collect or use personal information about a user (e.g., user data, information about the user's social network, the user's location and the time when the user is at that location, the user's biometric information, the user's activity and demographic information), the user is given one or more opportunities to regulate whether information may be collected, whether the personal information may be stored, whether the personal information may be used, and how information about the user is collected, stored, and used. That is, the systems and methods described herein specifically do so upon receiving explicit permission from the relevant user to collect, store, and / or use user personal information.
[0150] For example, a particular user may control whether a program or feature collects user information about that user or about other users associated with that program or feature. The user is presented with one or more options, such as options that allow the user to regulate the collection of information relevant to that user, options that allow the user to give permission or approval as to whether information may be collected, and what portions of the information may be collected. For example, the user may be given one or more such regulation options over a communications network. In addition, certain data may be treated in one or more ways before being stored or used such that personally identifiable information is removed. As one example, the identity of the user may be treated such that personally identifiable information cannot be determined. As another example, the geographic location of the user device may be generalized to a larger region such that the user's specific location cannot be determined.
[0151] However, as will be appreciated by those skilled in the art, the functional blocks, operations, features, methods, devices, and systems described in this disclosure may be integrated or divided into different combinations of systems, devices, and functional blocks. Any suitable programming language and programming techniques may be used to implement the routines of a particular embodiment. Different programming techniques, such as procedural or object-oriented, may be used. The routines may be executed on one processing device or multiple processors. Although steps, operations, or computations are presented in a particular order, this order may be changed in different particular embodiments. In some embodiments, multiple steps or operations shown in a sequence in this specification may be performed simultaneously.
Claims
1. 1. A computer-implemented method comprising: determining a type of object and a degree of movement of the object in a video sequence including a plurality of frames; and calculating a motion score of the video based on the type of the subject and the degree of movement of the subject, the calculating the motion score comprising: determining whether the extent of the movement of the object satisfies a threshold level of movement, the threshold level of movement being based on the type of the object, and calculating the motion score further comprising: determining whether the object type is landscape or inanimate if the extent of the object movement is determined to satisfy the threshold level of movement, and setting the motion score to a value that does not satisfy an indication threshold if the object type is determined to be landscape or inanimate; if it is determined that the extent of movement of the object does not meet the threshold level of movement, setting the motion score to a value that does not meet the display threshold; determining whether the motion score satisfies the display threshold after the calculating the motion score step; if the motion score satisfies the display threshold, displaying the plurality of frames of the motion image in succession, thereby displaying the motion image with subject motion; and if the motion score does not meet the display threshold, causing the motion image to be displayed without subject motion by displaying a particular frame of the plurality of frames of the motion image.
2. 2. The computer-implemented method of claim 1, wherein the motion image is displayed in a user interface including one or more other images, and if the motion score meets the display threshold, a size of the motion image in the user interface is larger than at least one of the one or more other images.
3. The motion image is displayed as part of a user interface that includes one or more other images, the method further comprising: determining a motion score for each of the one or more other images; and setting the display threshold based on the respective motion scores of the one or more other images.
4. A computer-implemented method according to any one of claims 1 to 3, wherein the step of calculating the motion score includes a step of applying a machine learning model trained to determine the motion score.
5. The video is captured by a client device and stored in a volatile memory of the client device; determining that the motion score does not satisfy a storage threshold that is different from the display threshold; and in response to determining that the motion score does not satisfy the storage threshold, acquiring a still image including only the particular frame of the plurality of frames of the motion image and storing the still image in a non-volatile memory of the client device.
6. A computer implemented method as described in claim 5, wherein the step of calculating the motion score includes a step of determining whether a user of the client device knows the subject.
7. A method implemented by a computer according to any one of claims 1 to 6, wherein the video includes at least one of a label indicating the type of subject and sounds recorded while capturing the video.
8. 1. A device for displaying a moving image composed of a plurality of frames, the device comprising: A processor; and a non-transitory computer-readable recording medium coupled to the processor and having a program stored thereon, the program, when executed by the processor, causing the processor to perform operations, the operations including: determining a type of object and a degree of movement of the object in a video sequence including a plurality of frames; calculating a motion score for the video based on the type of the object and the degree of movement of the object; and determining a respective motion score for one or more other images, wherein the motion images are displayed as part of a user interface that includes the one or more other images, the operations further comprising: setting a display threshold based on the respective motion scores of the one or more other images; determining whether the motion score satisfies the display threshold; displaying the motion image on the user interface with a subject motion by successively displaying the plurality of frames of the motion image if the motion score satisfies the display threshold; and if the motion score does not meet the display threshold, displaying a particular frame of the plurality of frames of the motion image on the user interface without subject motion.
9. The operation of calculating the motion score includes: determining whether the degree of movement of the object satisfies a threshold level of movement; determining whether the object type is a landscape or an inanimate object if the extent of the object movement is determined to satisfy the threshold level of movement, and setting the motion score to a value that does not satisfy the display threshold if the object type is determined to be a landscape or an inanimate object; and setting the motion score to a value that does not satisfy the display threshold if the extent of the movement of the object is determined to not satisfy the threshold level of movement.
10. The device of claim 9 , wherein the threshold level of movement is based on the type of the object.
11. a volatile memory coupled to the processor; and a non-volatile memory coupled to the processor, wherein the video images are captured by the device and stored in the volatile memory as they are captured, and the operations further comprise: determining that the motion score does not satisfy a storage threshold that is distinct from the display threshold; A device as described in any one of claims 8 to 10, comprising: in response to determining that the motion score does not satisfy the storage threshold, acquiring a still image including only the particular frame of the plurality of frames of the motion image and storing it in the non-volatile memory of the device.
12. A device described in any one of claims 8 to 11, wherein calculating the motion score includes a step of determining whether a user of the device knows the subject.
13. A device described in any one of claims 8 to 12, wherein the video includes at least one of a label indicating the type of subject and a sound recorded while capturing the video.
14. A program that, when executed by a processor, causes the processor to perform operations, the operations including: determining a type of object and a degree of movement of the object in a video sequence including a plurality of frames; calculating a motion score of the video based on the type of the subject and the degree of movement of the subject, wherein calculating the motion score includes: determining whether the extent of the movement of the object satisfies a threshold level of movement, the threshold level of movement being based on the type of the object; and calculating the motion score further comprising: determining whether the object type is a landscape or an inanimate object if the extent of the object movement is determined to satisfy the threshold level of movement, and setting the motion score to a value not satisfying an indication threshold if the object type is determined to be a landscape or an inanimate object; setting the motion score to a value that does not satisfy the display threshold if it is determined that the extent of movement of the object does not satisfy the threshold level of movement; determining whether the motion score satisfies the display threshold after calculating the motion score; if the motion score satisfies the display threshold, displaying the plurality of frames of the motion image in succession, thereby displaying the motion image with subject motion; If the motion score does not satisfy the display threshold, a particular frame of the plurality of frames of the motion image is displayed, thereby displaying the motion image without subject motion.
15. 15. The program of claim 14, wherein the motion image is displayed in a user interface including one or more other images, and when the motion score satisfies the display threshold, a size of the motion image in the user interface is larger than at least one of the one or more other images.
16. The motion image is displayed as part of a user interface that includes one or more other images, and the action further comprises: determining a motion score for each of the one or more other images; and and setting the display threshold based on the respective motion scores of the one or more other images.
17. A program described in any one of claims 14 to 16, wherein calculating the motion score includes a step of determining whether a user of a device including the processor is aware of the subject.
18. A program described in any one of claims 14 to 17, wherein the video includes at least one of a label indicating the type of subject and a sound recorded while shooting the video.
Citation Information
Patent Citations
Video camera switching device
JP1996317378A
Monitor
JP2000115762A
Video display device and video display method
JP2004357054A
Image reproducing apparatus and program
JP2006222510A
Data processing apparatus, data processing method, and program
JP2007221495A
