Method and system for identifying static video

Abstract

Some embodiments may include a computer-implemented method and / or system for identifying a static video. A method may include receiving an input video including a frame; preprocessing the frame of the input video; estimating local motion; compensating for global motion; calculating an object motion score based on entropy of the local motion estimate that has been adjusted by global motion compensation; normalizing the object motion score; determining whether the input video is a static video; and in response to the input video being a static video, providing the frame to a user instead of the input video.

Description

[0001] This application is a divisional application of the following application:

[0002] International application number of the original application: PCT / US2019 / 021442

[0003] International filing date of the original application: March 8, 2019

[0004] The national application number of the original application is: 201980005872.8,

[0005] The title of the invention of the original application: static video recognition. Background Art

[0006] Some mobile devices may include the ability to capture both video and still images. In some cases, a video may be captured that records images that occur just before and after a shutter press is detected to capture a still image. Such a video may be interesting and helpful in capturing images of subjects that a single still image may not contain. However, a video of a static or generally static scene (such as a building or landscape) may be detrimental to the user and may not be worth storing in memory on the device or in external memory. There are technical difficulties in distinguishing between static videos and videos with movement. For example, if an image is captured using a mobile device, the user's hands may be shaking so much that it may appear as if the video includes movement. Therefore, it may be useful to identify that a video is static (e.g., recording a relatively static scene or object).

[0007] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the inventor of the present invention is described in this background section, as well as various aspects of the description that may not be prior art at the time of filing, which are neither explicitly nor implicitly admitted as prior art of the present disclosure. Summary of the invention

[0008] Some embodiments relate generally to mobile device image capture, and more particularly to methods and systems for recognizing still video.

[0009] Some embodiments may include a computer-implemented method. The method may include acquiring an input video frame from one or more video frames of an input video, wherein the video frame is acquired before receiving a shutter press indication and after receiving the shutter press indication. The method also includes downsampling the input video frame to generate a downsampled video frame. The method also includes smoothing the downsampled video frame to generate a smoothed video frame. The method also includes calculating a temporal gradient and a spatial gradient of the smoothed video frame. The method also includes estimating local motion by calculating a local motion vector corresponding to a corresponding candidate pixel block from one or more candidate pixel blocks within the smoothed video frame, wherein the local motion vector is based on the spatial gradient and the temporal gradient. The method also includes estimating global motion by calculating a global motion vector corresponding to the smoothed video frame. The method also includes calculating one or more object motion vectors based on the corresponding local motion vectors and the global motion vector. The method also includes converting the one or more object motion vectors into a score map. The method also includes calculating an entropy score based on the score map. The method also includes mapping the entropy score to an object motion score corresponding to the smoothed video frame. The method further includes identifying the number of static frames based on the object motion score. The method further includes determining that the input video is a static video if a ratio of the number of static frames to the number of video frames satisfies a static frame threshold, and determining that the input video is a motion video if the ratio does not satisfy the static frame threshold. The method further includes performing a first operation if the input video is identified as the static video. The method further includes performing a second operation if the input video is identified as the motion video.

[0010] The method may also include converting the downsampled video frame from color to grayscale before smoothing. In some embodiments, calculating the spatial gradient and the temporal gradient of the smoothed video frame includes: marking underexposed pixels in the smoothed video frame and overexposed pixels in the smoothed video frame as invalid; and for valid pixels within the smoothed video frame: calculating the temporal gradient; calculating the spatial gradient in the x direction; and calculating the spatial gradient in the y direction. In some embodiments, the one or more candidate pixel blocks are determined by the following steps: selecting one or more grid points within the smoothed video frame as reference points for motion estimation; selecting a pixel block width and a pixel block height for a pixel block corresponding to each of the one or more grid points; calculating statistical information for each pixel block; determining whether the pixel block has a texture that satisfies a threshold texture value; and if the pixel block has a texture that satisfies the threshold texture value, identifying the pixel block as a candidate pixel block for motion estimation. In some embodiments, calculating the local motion vector includes calculating a local x motion value and calculating a local y motion value based on the spatial gradient and the temporal gradient of pixels in the candidate pixel block. In some embodiments, calculating the global motion vector includes calculating the global motion vector based on an average of valid local motion vectors. In some embodiments, calculating the one or more object motion vectors includes subtracting the global motion vector from the corresponding local motion vector. In some embodiments, converting the one or more object motion vectors into the score map includes normalizing the magnitude of the corresponding object motion vectors of the pixels within the smoothed video frame. In some embodiments, calculating the object motion score that is inversely proportional to the entropy score with respect to the score map includes assigning a higher score to the smoothed video frame when the motion is localized, and assigning a lower score to the smoothed video frame when the motion is global within the smoothed video frame. In some embodiments, the first operation includes marking the input video as static, retaining a shutter image corresponding to the input video in a memory, and deleting the input video from the memory. In some embodiments, the second operation includes retaining the shutter image corresponding to the input video and the input video in a memory.

[0011] Some embodiments may include a system. The system may include one or more processors; and a memory having instructions stored thereon, the instructions causing the one or more processors to perform operations when executed by the one or more processors, the operations including: obtaining an input video frame from one or more video frames of an input video, wherein the video frame is obtained before and after a shutter press indication is received; downsampling the input video frame to generate a downsampled video frame; smoothing the downsampled video frame to generate a smoothed video frame; calculating a temporal gradient and a spatial gradient of the smoothed video frame; estimating local motion by calculating a local motion vector corresponding to a corresponding candidate pixel block from one or more candidate pixel blocks within the smoothed video frame, wherein the local motion vector is based on the spatial gradient and the temporal gradient; estimating global motion by calculating a global motion vector corresponding to the smoothed video frame; calculating one or more object motion vectors based on the corresponding local motion vectors and the global motion vector; converting the one or more object motion vectors into a score map (score map) map); calculating an entropy score based on the score map; mapping the entropy score to an object motion score corresponding to the smoothed video frame; identifying the number of static frames based on the object motion score; determining that the input video is a static video if a ratio of the number of static frames to the number of video frames satisfies a static frame threshold, and determining that the input video is a motion video if the ratio does not satisfy the static frame threshold; performing a first operation if the input video is identified as the static video; and performing a second operation if the input video is identified as the motion video.

[0012] In some embodiments, the instructions further cause the one or more processors to perform operations including converting the downsampled video frame from color to grayscale prior to smoothing. In some embodiments, calculating the spatial gradient and the temporal gradient of the smoothed video frame includes: marking underexposed pixels in the smoothed video frame and overexposed pixels in the smoothed video frame as invalid; and for valid pixels within the smoothed video frame: calculating temporal gradients; calculating spatial gradients in the x direction; and calculating spatial gradients in the y direction. In some embodiments, the one or more candidate pixel blocks are determined by the following steps: selecting one or more grid points within the smoothed video frame as reference points for motion estimation; selecting a pixel block width and a pixel block height for a pixel block corresponding to each of the one or more grid points; calculating statistical information for each pixel block; determining whether the pixel block has a texture that satisfies a threshold texture value; and if the pixel block has a texture that satisfies the threshold texture value, identifying the pixel block as a candidate pixel block for motion estimation.

[0013] Some embodiments may include a non-transitory computer-readable medium having instructions that, when executed by one or more computers, cause the one or more computers to perform operations, the operations including: obtaining an input video frame from one or more video frames of an input video, wherein the video frame is obtained before and after receiving a shutter press indication; downsampling the input video frame to generate a downsampled video frame; smoothing the downsampled video frame to generate a smoothed video frame; calculating a temporal gradient and a spatial gradient of the smoothed video frame; estimating local motion by calculating a local motion vector, the local motion vector corresponding to a corresponding candidate pixel block from one or more candidate pixel blocks within the smoothed video frame, wherein the local motion vector is based on the spatial gradient and the temporal gradient; estimating global motion by calculating a global motion vector corresponding to the smoothed video frame; calculating one or more object motion vectors based on the corresponding local motion vectors and the global motion vector; converting the one or more object motion vectors into a score map (score map) map); calculating an entropy score based on the score map; mapping the entropy score to an object motion score corresponding to the smoothed video frame; identifying the number of static frames based on the object motion score; determining that the input video is a static video if a ratio of the number of static frames to the number of video frames satisfies a static frame threshold, and determining that the input video is a motion video if the ratio does not satisfy the static frame threshold; performing a first operation if the input video is identified as the static video; and performing a second operation if the input video is identified as the motion video.

[0014] In some embodiments, the operation further comprises converting the downsampled video frame from color to grayscale prior to smoothing. In some embodiments, calculating the spatial gradient and the temporal gradient of the smoothed video frame comprises: marking underexposed pixels in the smoothed video frame and overexposed pixels in the smoothed video frame as invalid; and for valid pixels within the smoothed video frame: calculating temporal gradients; calculating spatial gradients in the x direction; and calculating spatial gradients in the y direction. In some embodiments, the one or more candidate pixel blocks are determined by the following steps: selecting one or more grid points within the smoothed video frame as reference points for motion estimation; selecting a pixel block width and a pixel block height for a pixel block corresponding to each of the one or more grid points; calculating statistical information for each pixel block; determining whether the pixel block has a texture that satisfies a threshold texture value; and if the pixel block has a texture that satisfies the threshold texture value, identifying the pixel block as a candidate pixel block for motion estimation. In some embodiments, calculating the local motion vector comprises calculating a local x motion value and calculating a local y motion value based on the spatial gradient and the temporal gradient of pixels in the candidate pixel block. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of an example system and network environment that may be used with one or more implementations described herein;

[0016] Figure 2 is a flow chart illustrating an example method of identifying a static video according to some implementations;

[0017] Figure 3 is a diagram of an example video and shutter press timeline according to some implementations;

[0018] Figure 4 is a flow chart illustrating an example method of identifying static video according to some implementations; and

[0019] Figure 5 is a block diagram of an example device that may be used with one or more implementations described herein. DETAILED DESCRIPTION

[0020] The systems and methods provided herein can overcome one or more defects of some conventional imaging systems and methods. For example, an imaging device (e.g., a smart phone, etc.) allows a user to capture video and still images with a single shutter press event. Some embodiments of the systems and methods provided herein can identify static videos (e.g., videos of static scenes or static objects), and take actions based on the identification of static videos (e.g., deleting or removing static videos from memory), which can help reduce memory usage, communication bandwidth usage, and / or processor cycle usage. In particular, some embodiments can help compensate for global motion in image acquisition (e.g., hand tremors in a smart phone or other device during image acquisition), and emphasize localized object motion (e.g., children playing) by using a scoring system based on entropy calculations, and help weaken widespread general motion (e.g., leaves moving in the wind), thereby reducing the detection of motion when motion is spread across the frame. Some traditional imaging systems may not provide an effective and efficient way to identify static videos, where effective and efficient identification of static videos may be useful to users.

[0021] Figure 1 1 shows a block diagram of an exemplary network environment 100 that can be used in some embodiments described herein. In some embodiments, the network environment 100 includes one or more server systems, such as Figure 1 102 in the example of FIG. 104. For example, the server system 102 can communicate with a network 130. The server system 102 can include a server device 104 and a database 106 or other storage device. The network environment 100 can also include one or more client devices, such as client devices 120, 122, 124, and 126, which can communicate with each other and / or with the server system 102 via the network 130. The network 130 can be any type of communication network, including one or more of the Internet, a local area network (LAN), a wireless network, a switch or hub connection, etc. In some embodiments, the network 130 can include peer-to-peer communications 132 between devices, for example, using a peer-to-peer wireless protocol.

[0022] For ease of explanation, Figure 1One block is shown for server system 102, server device 104, and database 106, and four blocks are shown for client devices 120, 122, 124, and 126. The blocks representing server systems 102, 104, and 106 may represent multiple systems, server devices, and network databases, and the blocks may be provided in a configuration different from that shown. For example, server system 102 may represent multiple server systems that may communicate with other server systems via network 130. In some examples, database 106 and / or other storage devices may be provided in a server system block that is separate from server device 104 and may communicate with server device 104 and other server systems via network 130. Likewise, there may be any number of client devices.

[0023] Each client device may be any type of electronic device, such as a desktop computer, a laptop computer, a portable or mobile device, a camera, a cellular phone, a smart phone, a tablet computer, a television, a television set-top box or entertainment device, a wearable device (e.g., display glasses or goggles, a head-mounted display (HMD), a watch, a headset, an armband, jewelry, etc.), a virtual reality (VR) and / or augmented reality (AR) enabled device, a personal digital assistant (PDA), a media player, a gaming device, etc. Some client devices may also have a local database similar to database 106 or other storage. In other embodiments, the network environment 100 may not have all of the components shown, and / or may have other elements including other types of elements instead of or in addition to those described herein.

[0024] In various implementations, end users U1, U2, U3, and U4 may comprise one or more participants in a session and may communicate with server system 102 and / or with each other using respective client devices 120, 122, 124, and 126. In some examples, users U1, U2, U3, and U4 may interact with each other via applications running on respective client devices and / or server system 102 and / or via network services (e.g., image sharing services, messaging services, social networking services, or other types of network services) implemented on server system 102. For example, respective client devices 120, 122, 124, and 126 may transmit data to and from one or more server systems (e.g., server system 102).

[0025] In some embodiments, the server system 102 can provide appropriate data to the client device so that each client device can receive the communication content or shared content uploaded to the server system 102 and / or the network service. In some examples, users can interact via audio or video conferencing, audio, video or text chat, or other communication modes or applications. In some examples, the network service can include allowing users to perform various communications, form links and associations, upload and publish shared content (such as images, image synthesis (for example, albums including one or more images, image collages, videos, etc.), audio data and other types of content), receive various forms of data, and / or perform social-related functions. For example, the network service can allow users to send messages to specific or multiple other users, form social links in the form of association with other users in the network service, group other users in user lists, friend lists or other user groups, publish or send content including text, images, image synthesis, audio sequences or records, or other types of content for access by a designated user group of the network service, participate in live video, audio and / or text video conferencing or chat with other users of the service, etc. In some implementations, a "user" may include one or more programs or virtual entities, as well as humans interfacing with a system or network.

[0026] The user interface may allow for the display of images, image compositions, data, and other content as well as communications, privacy settings, notifications, and other data on the client devices 120, 122, 124, and 126 (or on the server system 102). Such an interface may be displayed using software on the client devices, software on the server devices, and / or a combination of client software and server software executed on the server device 104 (e.g., application software or client software that communicates with the server system 102). The user interface may be displayed by a display device of the client device or server device (e.g., a display screen, a projector, etc.). In some embodiments, an application running on the server system may communicate with the client device to receive user input at the client device and output data such as visual data, audio data, etc. at the client device.

[0027] Various implementations of the features described herein can use any type of system and / or service. For example, a social networking service, an image collection and sharing service, an auxiliary messaging service, or other network service (e.g., connected to the Internet) can include one or more of the described features accessed by client and server devices. Any type of electronic device can utilize the features described herein. Some embodiments can provide one or more features described herein on a client or server device that is disconnected from or intermittently connected to a computer network. In some examples, a client device that includes or is connected to a display device can examine and display images stored on a storage device local to the client device (e.g., not connected via a communication network), and can provide the features and results described herein that are visible to a user.

[0028] Figure 2 is a flow chart illustrating an example method 200 (e.g., a computer-implemented method) for identifying static video in accordance with some embodiments. Static video may include video in which a given percentage of frames of the video is determined to be static, as described herein. Static video may also include video that is static at certain time portions of the video and in which motion may occur at other time portions (e.g., a video is static from the beginning to near the end of the video when an object moves or when a device capturing an image is lowered or moved, etc.). Alternatively, according to particular embodiments, a video in which motion occurs at certain portions of the video (e.g., the beginning, middle, or end) may be identified as a motion video or a static video. In general, parameters for identifying static video may be added, adjusted, configured, or removed based on the intended application or implementation of the subject matter disclosed herein. Such parameters may include thresholds for local motion estimation, thresholds for object motion scores, thresholds for the ratio of frames having object motion scores that meet a threshold to other frames of the video, and the like. Reference is made below to Figure 4 Thresholds are discussed in more detail.

[0029] In some embodiments, method 200 (and method 400) may be performed, for example, in Figure 1 In other embodiments, some or all of the method 200 may be implemented on the server system 102 shown in FIG. Figure 1 The method 200 may be implemented on one or more client devices 120, 122, 124, or 126, one or more server devices, and / or both server devices and client devices. In the described examples, the implemented system includes one or more digital hardware processors or processing circuits ("processors"), and one or more storage devices (e.g., database 106 or other memory). In some embodiments, different components of one or more servers and / or clients may perform different blocks or other portions of the method 200.

[0030] Some embodiments may initiate method 200 based on user input and / or device context (obtained with the user's permission). For example, the user may select the initiation of method 200 from a displayed user interface. In some embodiments, method 200 or portions thereof may be performed by the user through the guidance of user input. For example, some embodiments may include presenting an option to the user to enable or disable static video recognition. Without explicit permission from the user, the system may not use, process or store user information, such as video and image information, etc.

[0031] In some embodiments, method 200 may be automatically invoked (or invoked automatically with the user's permission). For example, identification of still videos may be automatically performed when a user takes a photo, and may be used to determine whether to present a video associated with the photo to the user (e.g., if the video is identified as still, the video may not be presented or stored, and if the video is determined to be motion video, the video may be stored and presented to the user. For example, when the user views a photo associated with the video).

[0032] In some embodiments, method 200 or a portion of the method may be automatically initiated by a device. For example, the method (or a portion thereof) may be executed periodically or based on the occurrence of one or more specific events or conditions. For example, such events or conditions may include a given number of photos acquired by a user, received by a device (e.g., a user device), uploaded to a device (e.g., a user device) or otherwise accessible by a device (e.g., a user device), a predetermined period of time that has passed since the last execution of method 200, and / or one or more other events or conditions that may be specified in the settings of a device implementing method 200. In some embodiments, such conditions may be specified in advance by a user in a stored user customization preference (which may be accessed by a device or method with the user's consent). In some examples, a device (server or client) may execute method 200 by accessing one or more applications that receive electronic conversation messages (if user consent is received). In another example, a camera, a cellular phone, a tablet computer, a wearable device or other client device may acquire video and still images and may execute method 200. Additionally, or alternatively, the client device may send one or more videos to the server over the network, and the server may process the videos using method 200. The above discussion regarding method 200 also applies to method 400.

[0033] At 202, an input video including frames is received. For example, the video may be captured by a client device (e.g., a mobile device) associated with a user. The video may be captured in response to a shutter press of a camera or other imaging device to capture frames. For example, a user may open a camera application on a smartphone or other mobile device. The camera application may begin acquiring video images before receiving a shutter press indication, and continue capturing video after receiving the shutter press indication. Thus, the video captures the moments before and after the shutter press within the camera application executed on the smartphone. Processing continues to 204.

[0034] For example, Figure 3 As shown, video 302 may include frames (304, 306) captured before shutter press indication 310 and frames (308) captured after shutter press indication 310. Video may also be captured in a similar manner within a camera or other image acquisition device. Pre-processing may include down-sampling, conversion to grayscale, and / or smoothing.

[0035] At 204, the frames of the input video are preprocessed. Figure 4 404-408 describe the details of the exemplary pre-processing. Processing continues to 206.

[0036] At 206, local motion is estimated. Local motion includes movement that is part of a subset of frames of the video, such as motion associated with leaf shake. For example, a local motion vector may be calculated that represents an estimate of local motion within one or more pixel blocks (i.e., pixel segments or groups) of a video frame. Figure 4 Details of exemplary local motion estimation are described in 410 - 424 of FIG.

[0037] At 208, global motion is compensated. Global motion can include motion that occurs in all or most frames of the video. Global motion can be caused by hand motion. For example, a user taking a picture with a smart phone may have some hand movement before, during, and / or after capturing the image due to hand tremors, accidentally dropping the smart phone, etc. The user's hand movement can appear as global motion within the video, which covers the time period from before the shutter is pressed to after the shutter is pressed. Global motion compensation seeks to reduce or eliminate the effects of global motion to allow the system to identify static video more accurately than if global motion artifacts were not compensated. Figure 4 Details of exemplary global motion compensation are described at 426-428 of FIG.

[0038] In block 210, an object motion score is calculated based on the entropy of the local motion estimate that has been adjusted by global motion compensation. Figure 4Details of the entropy-based object motion score calculation are described at 430-432. Processing continues to 212.

[0039] At 212 , the object motion score is normalized. Processing continues to 214 .

[0040] At 214, it is determined whether the input video is a static video. The determination may include identifying a video of a static scene or object using a system or method as described herein and providing an indication of a video of a static scene or object (or a motion video when no static video is identified). Based on the determination, processing continues to 216 or 218.

[0041] If the input video is not a static video, then at 216, both the input video and the frame are provided to the user. In some embodiments, both the input video and the frame are included in a memory. If the input video is a static video, then at 218, the frame is provided to the user instead of the input video. In some embodiments, the frame is stored in a memory instead of the input video.

[0042] 2 and 4, the various blocks (e.g., blocks 202-218 and 402-440, respectively) are shown as being executed sequentially. However, it should be understood that one or more of these blocks may be rearranged to accommodate a particular implementation, and that these blocks or portions thereof may be executed simultaneously in some implementations. It should also be understood that in some examples, various blocks may be eliminated, divided into additional blocks, and / or combined with other blocks. The table may then be used to determine a threshold based on the values ​​in the table.

[0043] Figure 4 is a flow chart illustrating an example method of identifying a static video according to some embodiments. Figure 4 402 and 412 are data or data storage and are shown in dashed lines to help illustrate the method.

[0044] In operation, video frames are obtained 402. An input video frame may be obtained from one or more video frames of a video, wherein the video frames are obtained before receiving a shutter press indication and after receiving a shutter press indication. For example, the video frames may be captured a predetermined amount of time before and after the shutter press indication, such as two seconds before and after the shutter press indication.

[0045] At 404 , the video frame is downsampled. Downsampling may include reducing the size of the video frame (file size, frame size, video resolution, etc.) to improve processing time, reduce memory usage, etc. Processing continues at 406 .

[0046] At 406, the video frame is converted to grayscale. Because color may not be necessary to recognize static video, converting to grayscale can help speed up processing time and reduce memory usage. Processing continues to 408.

[0047] At 408, the downsampled grayscale image is smoothed as described below.

[0048] 404-408 may include preprocessing. For example, the input video frame 402 may have a high resolution, such as 4032×3024. To reduce computation, the input video frame 402 is first downsampled to a much smaller frame I, such as 64×48. In some cases, the input video frame may be in a Bayer red, green, and blue (RGB) format or a YUV format. Grayscale frames may be used to detect whether there is motion, so color video frames may be converted to grayscale frames. For example, a 64×48 Bayer frame is obtained and downsampled. The Bayer frame may be converted to a grayscale image by averaging the R, Gr, Gb, and B channels. In order to quickly convert 12 / 14 / 16-bit Bayer data to 8 bits and maintain dynamic range, the average of the Bayer frame is calculated and the pixels are scaled by desired_average / average_value, where desired_average is a manually selected value, for example, desired_average=128.

[0049] In order to improve the robustness of the technique to image noise, the input frame I may be smoothed by a Gaussian filter and the resulting frame is denoted as I_smoothed. The Gaussian smoothed previous frame is denoted as I_smoothed_previous.

[0050] After pre-processing, local motion is estimated, for example, in blocks 410 - 424 .

[0051] At 410, temporal gradients are calculated using the previous frame from frame buffer 412 and the current video frame 402. At 414, spatial gradients are calculated using the current video frame.

[0052] In some embodiments, in order to speed up local motion estimation, the spatial gradient and temporal gradient of the input frame may be pre-calculated. For example, the spatial gradient and temporal gradient may be calculated as follows:

[0053] All overexposed pixels (eg, pixels with intensities above a threshold (eg, “over_expose_threshold”)) and underexposed pixels (eg, pixels with intensities below a threshold (eg, “under_expose_threshold”)) are marked as invalid pixels.

[0054] Valid(x,y)=1, if under_expose_threshold<=I_smoothed(x,y)<=over_expose_threshold

[0055] and

[0056] under_expose_threshold <I_smoothed_previous(x,y)<over_expose_threshold

[0057] and

[0058] under_expose_threshold <I_smoothed_previous(x+1,y)<over_expose_threshold

[0059] and

[0060] under_expose_threshold <I_smoothed_previous(x-1,y)<over_expose_threshold

[0061] and

[0062] under_expose_threshold <I_smoothed_previous(x,y+1)<over_expose_threshold

[0063] and

[0064] under_expose_threshold <I_smoothed_previous(x,y-1)<over_expose_threshold

[0065] For others, Valid(x, y) = 0

[0066] For valid pixels where Valid(x,y)=1, the three gradients I_t(x,y), I_x(x,y), and I_y(x,y) are calculated as follows:

[0067] The temporal gradient, ie, the pixel difference between I_smoothed and I_smoothed_previous, is expressed as I_t(x, y) = I_smoothed(x, y) - I_smoothed_previous(x, y).

[0068] The spatial gradient of I_smoothed along the x direction is expressed as

[0069] I_x(x,y)=(I_smoothed(x+1,y)-I_smoothed(x-1,y)) / 2.

[0070] The spatial gradient of I_smoothed along the y direction is expressed as

[0071] I_y(x,y)=(I_smoothed(x,y+1)-I_smoothed(x,y-1)) / 2.

[0072] For invalid pixels with Valid(x,y)=0, set I_t(x,y)=I_x(x,y)=I_y(x,y)=0.

[0073] Processing continues to 416.

[0074] At 416, a grid of points is generated, and at 418, a pixel block is determined for each grid point.

[0075] For example, a grid of points p(i,j) within a frame is calculated for local motion estimation, where i is between 1 and the grid width and j is between 1 and the grid height. When calculating the grid, a certain percentage of margin on each side can be excluded, such as 6.25%. The grid points can be evenly spaced, such as 4 pixels along the x direction and 4 pixels along the y direction. For example, if the frame size is 64×48, 64×6.25%=4 pixels are excluded on four sides, leaving a grid with grid width=15 and grid height=11.

[0076] Processing continues to 420 .

[0077] At 420, statistical information is calculated for each pixel block. For example:

[0078] For each point p(i,j) in the grid, get a patch of patchWidth x patchheight (eg, patchWidth = patchHeight = 7) from I_smoothed.

[0079] First collect a set of statistics for this block:

[0080] Ixx is the average value of I_x(x,y)*I_x(x,y) of all pixels in the block.

[0081] Iyy is the average value of I_y(x,y)*I_y(x,y) of all pixels in the block.

[0082] Ixy is the average value of I_x(x,y)*I_y(x,y) of all pixels in the block.

[0083] 1xt is the average value of I_x(x,y)*I_t(x,y) of all pixels in the patch.

[0084] Iyt is the average value of I_y(x,y)*I_t(x,y) of all pixels in the block.

[0085] The calculation of statistics can be optimized by using integral images.

[0086] Processing continues to 422.

[0087] At 422, it is determined whether each block has a texture that satisfies a texture threshold. Checking the texture of the block may be helpful because it is useless to detect motion using a textureless block. Also, blocks on an edge may not be helpful to use because motion can be detected along the edge arbitrarily. As a result, if the texture does not satisfy the texture threshold, then in some embodiments, method 400 performs the steps starting at 402 until the texture satisfies the texture threshold. Checking for appropriate texture may include:

[0088] The Hessian matrix of the above block is a 2×2 matrix:

[0089] {Ixx Ixy

[0090] Ixy Iyy}

[0091] The determinant of this matrix is ​​denoted as det. This matrix has two eigenvalues. The larger one is denoted as the maximum eigenvalue (max-eigenvalue) and the smaller one is denoted as the minimum eigenvalue (min_eigenvalue).

[0092] A block must meet the following criteria to be considered sufficiently textured:

[0093] det>0 and

[0094] min_eigenvalue > eigenvalue threshold * frame width * frame height

[0095] and

[0096] max_eigenvalue<eigenvalue ratio threshold*min_eigenvalue

[0097] The feature value threshold can be adjusted manually, and a typical value is 0.0025.

[0098] The eigenvalue ratio threshold can also be adjusted manually, and a typical value is 5.

[0099] If a block does not pass the above conditional check, the motion vector of the block is set to unavailable and the block is excluded in the following steps.

[0100] motion_x = not available.

[0101] motion_y = not available.

[0102] For blocks having texture that meets the texture threshold, processing continues to 424 .

[0103] At 424, a local motion vector is calculated to estimate the local motion of the corresponding block. The local motion vector may correspond to a corresponding candidate pixel block from one or more candidate pixel blocks within the smoothed video frame, wherein the local motion vector is based on a spatial gradient and a temporal gradient. For example:

[0104] motion_x=(-Ixt*Iyy+Iyt*Ixy) / det

[0105] motion_y=(Ixt*Ixy-Iyt*Ixx) / det

[0106] Processing continues to 426.

[0107] At 426, a global motion vector is calculated to estimate global motion. In some embodiments, the global motion vector corresponds to a smoothed video frame. Because there are often small hand movements when a user captures photos or videos on a mobile device, estimating the global camera motion between the current frame and the previous frame is useful for compensating for global motion. Camera motion can be modeled as a translation vector [global_motion_x, global_motion_y] and estimated by averaging the valid motion vectors from the previous step. Alternatively, if more computation is available, an affine model or homography model can be used with random sample consensus (RANSAC) to more accurately estimate global motion. Processing continues to 428.

[0108] At 428, the object motion vector is calculated by subtracting the global motion vector from the local motion vector. For example:

[0109] The global motion is subtracted from the motion vector to obtain the subject motion vector [subject_motion_x, subject_motion_y] = [motion_x - global_motion_x, motion_y - global_motion_y].

[0110] Processing continues to 430 .

[0111] At 430, the score for each block is calculated and placed into a score map. For example:

[0112] The magnitude of the motion vector is converted into a score between 0 and 1 for each block to create a scoremap as follows:

[0113] score(i,j)=1-exp(-(subject_motion_x(i,j)*subject_motion_x(i,j)+

[0114] subject_motion_y(i,j)*subject_motion_y(i,j)) / motion_param.

[0115] motion_param (motion parameter) can be determined manually, and in some embodiments a typical value is 3.

[0116] Processing continues to 432.

[0117] At 432, an entropy-based object motion score is calculated for the video frame. The object motion score can be based on the corresponding local motion vector and the global motion vector. For example:

[0118] First, the entropy score of the score map is calculated as follows:

[0119] For all k and l, total_score = sum(score(k,l)+ε).

[0120] p(k,l)=(score(k,l)+ε) / total_score.

[0121] Entropy = -sum(Log[p(k,1)]*p(k,1))

[0122] ε can be a decimal to avoid problems caused by 0.

[0123] The entropy is then mapped to the subject_motion_score corresponding to the smoothed video frame using a non-linear mapping function, where the subject_motion_score is between 0 and 1:

[0124] subject_motion_score=1-exp(entropy*subject_motion_param1)*subject_motion_param2

[0125] Subject_motion_param1 and subject_motion_param2 are two parameters that are manually adjusted.

[0126] The object motion score is large when there is large object motion in the scene. It is very close to 0 when the scene is static.

[0127] Processing continues to 434.

[0128] At 434, the object motion scores of one or more frames are normalized. For example:

[0129] For each video, each frame (with frame number i) processed for static video recognition receives subject_motion_score(i). The maximum value of the subject motion score is expressed as:

[0130] For all i, subject_motion_score_max = max(subject_motion_score(i))

[0131] If subject_motion_score_i is not too small, normalize subject_motion_score by subject_motion_score_max. For example:

[0132] when

[0133] When normalized_subject_motion_score_i>normalized_subject_motion_min_threshold,

[0134] normalized_subject_motion_score_i=subject_motion_score_i /

[0135] subject_motion_score_max

[0136] Otherwise, normalized_subject_motion_score_i=subject_motion_score_i

[0137] Processing continues to 436.

[0138] At 436, the normalized object motion score is thresholded. For example:

[0139] Count the number of frames with low normalized object motion scores.

[0140] number_of_static_frames=count(normalized_subject_motion_score_i <normalized_subject_motion_score_threshold)。

[0141] Processing continues to 438.

[0142] At 438, the ratio of video frames identified as static to all frames in the video is compared to a static frame ratio threshold. For example:

[0143] If the number of static frames (number_of_static_frames) / (number_of_all_frames) of all frames > the static frame ratio threshold, the video is marked as a static video, otherwise the video is not marked as a static video.

[0144] Processing continues to 440 .

[0145] At 440, an action is taken based on whether the video is identified as a static video. For example, if the video is identified as static, one operation is performed, and if the video is identified as non-static, a different operation is performed. In some examples, when the video is identified as static, the video may be marked as static, a shutter image (or still image) corresponding to the video may be retained in a memory, and the video may be deleted from the memory. When the video is identified as non-static, both the shutter image (or still image) and the video may be retained in a memory (e.g., a memory), and in some examples, the video may be presented to the user, for example, displayed as a continuous frame on a display device of a user device or other device.

[0146] Figure 5 5 is a block diagram of an example device 500 that can be used to implement one or more features described herein. In one example, the device 500 can be used to implement a computer device, such as a server device (e.g., Figure 1 The device 500 may be a server device 104 of the embodiment of the present invention and perform appropriate method implementations described herein. The device 500 may be any suitable computer system, server or other electronic or hardware device. For example, the device 500 may be a mainframe computer, a desktop computer, a workstation, a portable computer or an electronic device (a portable device, a mobile device, a cellular phone, a smart phone, a tablet computer, a television, a television set-top box, a personal digital assistant (PDA), a media player, a gaming device, a wearable device, etc.). In some embodiments, the device 500 includes a processor 502, a memory 504 and an I / O interface 506.

[0147] Processor 502 may be one or more processors and / or processing circuits that execute program code and control the basic operations of device 500. A "processor" includes any suitable hardware and / or software system, mechanism, or component that processes data, signals, or other information. A processor may include a system with a general-purpose central processing unit (CPU), multiple processing units, dedicated circuits for implementing functions, or other systems. Processing need not be limited to a specific geographic location or have time constraints. For example, a processor may perform its functions in "real time", "offline", "batch mode", etc. Parts of processing may be performed by different (or the same) processing systems at different times and in different locations. A computer may be any processor that communicates with a memory.

[0148] Memory 504 is generally provided in device 500 for access by processor 502 and may be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc., suitable for storing instructions executed by the processor and located separately from and / or integrated with processor 502. Memory 504 may store software operated by processor 502 on server device 500, including operating system 508, one or more applications 510, such as still video recognition application 512, other applications 514, and application data 520. In some embodiments, application 510 may include software that enables processor 502 to perform the functions described herein (e.g., Figure 2 and / or some or all of the methods of 4).

[0149] Other applications 514 (or engines) may also or alternatively be included in the application 510, such as camera applications, email applications, SMS and other telephone communication applications, web browser applications, media display applications, communication applications, web hosting engines or applications, social network engines or applications, etc. Any software in the memory 504 may be stored alternatively in any other suitable storage location or computer-readable medium. In addition, the memory 504 (and / or other connected storage devices) can store application data, such as electronic conversation messages, electronic conversation message groups, conversation frame elements, user data and preferences, and other instructions and data used in the features described herein. The memory 504 and any other type of storage (disk, optical disk, tape or other tangible medium) can be considered as "storage" or "storage device". For example, the application data 520 may include images 522 and videos 524.

[0150] The I / O interface 506 can provide functions that enable the device 500 to interface with other systems and devices. For example, a network communication device, a storage device (e.g., a memory and / or a database 106) and an input / output device can communicate via the I / O interface 506. In some embodiments, the I / O interface can be connected to an interface device including an input device (keyboard, a pointing device, a touch screen, a microphone, a camera, a scanner, etc.) and / or an output device (a display device, a speaker device, a printer, a motor, a tactile output device, etc.). The audio input / output device 530 is an example of an input and output device that can be used to receive audio input and provide audio output (e.g., a voice interface output), as described herein. The audio input / output device 530 can be connected to the device 500 via a local connection (e.g., a wired bus, a wireless interface) and / or via a networked connection, and can be any suitable device, some examples of which are described below.

[0151] For ease of explanation, Figure 5 A block of each of processor 502, memory 504, I / O interface 506, and software blocks 508 and 510 is shown. These blocks may represent one or more processors or processing circuits, operating systems, memories, I / O interfaces, applications, and / or software modules. In other embodiments, device 500 may not have all of the components shown and / or may have other elements including other types of elements instead of, or in addition to, those shown here. Although server system 102 is described as performing the operations described in some embodiments herein, any suitable component or combination of components of server system 102 or similar systems, or any suitable processor or processors associated with such systems may perform the described operations.

[0152] The client device may also implement and / or be used with the features described herein, for example, Figure 1 500. An example client device may be a computer device that includes some components similar to device 500, such as a processor 502, a memory 504, and an I / O interface 506. An operating system, software, and applications suitable for the client device may be provided in the memory and used by the processor, such as image management software, client group communication application software, etc. The I / O interface for the client device may be connected to a network communication device and input and output devices, such as a camera for capturing images or video, an audio speaker device for outputting sound, a display device for outputting images or video, or other output devices.

[0153] One or more methods described herein (e.g., methods 200 and 400) may be implemented by computer program instructions or codes that can be executed on a computer. For example, the code may be implemented by one or more digital processors (e.g., microprocessors or other processing circuits) and may be stored on a computer program product that includes a non-transitory computer-readable medium (e.g., storage medium), such as a magnetic, optical, electromagnetic or semiconductor storage medium, including semiconductor or solid-state memory, tape, removable computer disk, random access memory (RAM), read-only memory (ROM), flash memory, rigid disk, optical disk, solid-state memory drive, etc. The program instructions may also be contained in an electronic signal and provided as an electronic signal, such as in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods may be implemented in hardware (logic gates, etc.) or in a combination of hardware and software. Example hardware may be a programmable processor (e.g., a field programmable gate array (FPGA), a complex programmable logic device), a general-purpose processor, a graphics processor, an application-specific integrated circuit (ASIC), etc. One or more of the methods may be performed as part or component of an application running on the system, or as an application or software running in conjunction with other applications and an operating system.

[0154] One or more of the methods described herein may be run in a standalone program that can be run on any type of computing device, a program running on a web browser, a mobile application ("app") running on a mobile computing device (e.g., a cellular phone, a smart phone, a tablet computer, a wearable device (a watch, an armband, jewelry, headwear, goggles, glasses, etc.), a laptop computer, etc.). In one example, a client / server architecture may be used, for example, a mobile computing device (as a client device) sends user input data to a server device, and receives final output data from the server for output (e.g., for display). In another example, all computations may be performed within a mobile application (and / or other applications) on the mobile computing device. In another example, computations may be split between the mobile computing device and one or more server devices.

[0155] Any software in the memory may alternatively be stored in any other suitable storage location or computer-readable medium. In addition, the memory (and / or other connected storage devices) may store one or more messages, one or more classifications, electronic encyclopedias, dictionaries, thesauri, knowledge bases, message data, grammars, user preferences, and / or other instructions and data used in the features described herein. The memory and any other type of storage (disk, optical disk, tape or other tangible medium) may be considered "storage" or "storage devices".

[0156] The I / O interface can provide the function that enables the server device to interface with other systems and devices. The interface device can be included as a part of the device, or can be separated and communicate with the device. For example, a network communication device, a storage device (e.g., a memory and / or database 106) and an input / output device can communicate through the I / O interface. In some embodiments, the I / O interface can be connected to an interface device, such as an input device (keyboard, a pointing device, a touch screen, a microphone, a camera, a scanner, a sensor, etc.) and / or an output device (a display device, a speaker device, a printer, a motor, etc.).

[0157] Some examples of interface devices that can be connected to an I / O interface can include one or more display devices, which can be used to display content, such as images, videos, and / or user interfaces of output applications described herein. Display devices can be connected to devices via local connections (e.g., display bus) and / or via networking connections, and can be any suitable display devices. Display devices can include any suitable display devices, such as LCD, LED, or plasma display screens, CRT, televisions, monitors, touch screens, 3-D display screens, or other visual display devices. For example, display devices can be flat display screens provided on mobile devices, multiple display screens provided in goggles or headphone devices, or monitor screens for computer devices.

[0158] The I / O interface can interface to other input and output devices. Some examples include one or more cameras or other imaging devices capable of capturing images. Some embodiments may provide a microphone for capturing sound (e.g., as part of a captured image, voice commands, etc.), an audio speaker device for outputting sound, or other input and output devices.

[0159] Although described with respect to specific embodiments, these specific embodiments are merely illustrative and not restrictive. The concepts shown in the examples can be applied to other examples and embodiments.

[0160] In certain embodiments discussed herein, where personal information about a user (e.g., user data, information about a user's social network, the user's location and the time the user is at that location, the user's biometric information, the user's activities and demographic information) may be collected or used, one or more opportunities are provided to the user to control whether the information is collected, whether the personal information is stored, whether the personal information is used, and how the information about the user is collected, stored and used. That is, the systems and methods discussed herein collect, store and / or use user personal information, particularly when explicit authorization is received from the relevant user. For example, the user is provided with control over whether a program or feature collects user information about the particular user or other users associated with the program or feature. One or more options are presented to each user for whom personal information is to be collected to allow control over the collection of information related to the user, thereby providing permission or authorization as to whether to collect information and as to which parts of the information are to be collected. For example, one or more such control options may be provided to the user via a communication network. In addition, before certain data is stored or used, the data may be processed in one or more ways to remove personally identifiable information. As an example, the identity of the user may be processed so that personally identifiable information cannot be determined. As another example, the user's geographic location may be generalized to a larger area so that the user's specific location cannot be determined.

[0161] Note that the functional blocks, operations, features, methods, devices and systems described in this disclosure may be integrated or divided into different combinations of systems, devices and functional blocks, as known to those skilled in the art. Any suitable programming language and programming techniques may be used to implement the routines of a particular embodiment. Different programming techniques may be employed, for example, procedural or object-oriented. The routines may be executed on a single processing device or on multiple processors. Although steps, operations or calculations may be presented in a particular order, the order may be changed in different particular embodiments. In some embodiments, multiple steps or operations shown as sequential in this specification may be performed simultaneously.

Claims

1. A method for identifying static video, characterized in that: include: receiving an input video comprising frames; Preprocessing the frames of the input video; Estimate local motion; Compensation for global motion; calculating an object motion score based on the entropy of the local motion estimate that has been adjusted by global motion compensation; normalizing the object motion score; Determine whether the input video is a static video; as well as In response to the input video being a static video, providing the frame to a user instead of the input video, The step of calculating the object motion score based on the entropy of the local motion estimation adjusted by global motion compensation comprises: Calculating the object motion vector by subtracting the global motion vector from the local motion vector; converting the object motion vector into a score map; calculating an entropy score for the score map; and mapping the entropy score into the object motion score, Wherein, the input video includes frames, and determining whether the input video is a static video includes: counting the number of frames having a low normalized object motion score compared to a threshold; the ratio of the number of frames identifying static to the number of all frames; and In response to the ratio of frames exceeding a static frame ratio threshold, it is determined that the input video is the static video.

2. The method according to claim 1, characterized in that Wherein compensating the global motion includes compensating the global motion occurring in at least some frames of the input video.

3. The method according to claim 1, characterized in that Determining whether the input video is a static video is based on the input video including a static scene or object indicated by the object motion score.

4. The method according to claim 1, characterized in that Also included is providing a frame and the input video to the user in response to the input video not being a static video.

5. The method according to claim 1, characterized in that The input video includes a current frame and a previous frame, and estimating the local motion includes: calculating a temporal gradient using the current frame and the previous frame from a frame buffer; Using the current frame to calculate a spatial gradient; Generate a grid of points; Determining a pixel block for each grid point; and The local motion is estimated based on the grid of points of the pixel block.

6. The method according to claim 1, characterized in that Compensating for global motion involves estimating the global motion by using an affine model or a homography model in concert with random samples.

7. The method according to claim 1, characterized in that The input video captures the moments before and after a shutter press indication is received.

8. A system for identifying static video, characterized in that: include: one or more processors; and A memory having instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform the following operations: receiving an input video comprising frames; Preprocessing the frames of the input video; Estimate local motion; Compensation for global motion; calculating an object motion score based on the entropy of the local motion estimate that has been adjusted by global motion compensation; normalizing the object motion score; Determine whether the input video is a static video; and In response to the input video being a static video, providing the frame to a user instead of the input video, The step of calculating the object motion score based on the entropy of the local motion estimation adjusted by global motion compensation comprises: Calculating the object motion vector by subtracting the global motion vector from the local motion vector; converting the object motion vector into a score map; calculating an entropy score for the score map; and mapping the entropy score into the object motion score, Wherein, the input video includes frames, and determining whether the input video is a static video includes: counting the number of frames having a low normalized object motion score compared to a threshold; the ratio of the number of frames identifying static to the number of all frames; and In response to the ratio of frames exceeding a static frame ratio threshold, it is determined that the input video is the static video.

9. The system according to claim 8, characterized in that Wherein compensating the global motion includes compensating the global motion occurring in at least some frames of the input video.

10. The system according to claim 8, characterized in that Determining whether the input video is a static video is based on the input video including a static scene or object indicated by the object motion score.

11. The system according to claim 8, characterized in that The operations also include providing a frame and the input video to the user in response to the input video not being a static video.

12. The system according to claim 8, characterized in that The input video includes a current frame and a previous frame, and estimating the local motion includes: calculating a temporal gradient using the current frame and the previous frame from a frame buffer; Calculating a spatial gradient using the current frame; Generate a grid of points; Determining a pixel block for each grid point; and The local motion is estimated based on the grid of points of the pixel block.

13. The system according to claim 8, characterized in that Compensating for global motion involves estimating the global motion by using an affine model or a homography model in concert with random samples.

14. A non-transitory computer-readable medium having instructions, characterized in that: The instructions, when executed by one or more computers, cause the one or more computers to perform operations including: receiving an input video comprising frames; Preprocessing the frames of the input video; Estimate local motion; Compensation for global motion; calculating an object motion score based on the entropy of the local motion estimate that has been adjusted by global motion compensation; normalizing the object motion score; Determining whether the input video is a static video; and In response to the input video being a static video, providing the frame to a user instead of the input video, The step of calculating the object motion score based on the entropy of the local motion estimation adjusted by global motion compensation comprises: Calculating the object motion vector by subtracting the global motion vector from the local motion vector; converting the object motion vector into a score map; calculating an entropy score for the score map; and mapping the entropy score into the object motion score, Wherein, the input video includes frames, and determining whether the input video is a static video includes: counting the number of frames having a low normalized object motion score compared to a threshold; the ratio of the number of frames identifying static to the number of all frames; and In response to the ratio of frames exceeding a static frame ratio threshold, it is determined that the input video is the static video.

15. The computer-readable medium of claim 14, wherein: Compensating the global motion includes compensating the global motion occurring in at least some frames of the input video.

16. The computer-readable medium of claim 14, wherein: Determining whether the input video is a static video is based on the input video including a static scene or object indicated by the object motion score.

Images