Intelligent time-lapse video capture on camera devices using neural network input to reduce the video bit rate to save bandwidth

Abstract

A device for implementing interval video shooting on a camera device by reducing the bit rate of the video to save bandwidth with the help of neural network input includes an interface and a processor. The interface can be configured to receive pixel data generated by a capture device. The processor can be configured to perform the following operations: perform computer vision operations on video frames to detect objects; perform classification of detected objects based on the characteristics of the objects; determine whether the classification of the objects corresponds to an event; generate a complete video stream in response to all video frames; and generate encoded video frames. The complete video frames can be recorded to a storage medium local to the device. The encoded video frames can be transmitted to a cloud service. The encoded video frames can include a first sample of the video frame and a second sample of the video frame, the first sample being selected at a first rate when the event is not detected, and the second sample being selected at a second rate when the event is detected.

Description

Technical Field

[0001] The present invention relates generally to computer vision, and more particularly to methods and / or apparatus for implementing time-lapse video on a camera device with the aid of a neural network input to save bandwidth by reducing the bit rate of the video. Background Art

[0002] The time-lapse video mode for conventional Internet-connected / cloud-enabled cameras is typically implemented by software operating on a cloud server. The software relies on the distributed processing resources (i.e., scalable computing) of the cloud server. The time-lapse video clips are displayed at a fixed frame rate to achieve a fast-forward effect. For example, video frames are selected at fixed intervals to create a time-lapse video (i.e., every 30th frame from a thirty frames per second video is selected to create a time-lapse video).

[0003] Internet-connected / cloud-enabled cameras encode video data and then transmit the encoded video stream to a cloud server. Many extra CPU cycles are used on the cloud server to decode a regular video stream (i.e., compressed video using AVC encoding or HEVC encoding), extract frames at fixed frame intervals, and then transcode those frames into timelapse video. At the same time, even with timelapse video, it is difficult for users to find important details. Because timelapse video always uses a fixed frame rate and does not use all of the captured video data, video frames at normal display speed are not available for the duration of the content of interest to the user in the captured video. For example, when a security camera observes a potential event of interest (e.g., a pedestrian), the timelapse will be the same as when the security camera does not observe any event of particular interest.

[0004] For residential security cameras, storage as a service (SaaS) cameras, or other types of video security cameras, video is continuously uploaded to a server (either locally or in the cloud). The continuous upload of data results in a large amount of data usage. Bandwidth consumption and storage usage are highly dependent on the average bit rate of the recorded video stream. Conventional methods for reducing bandwidth consumption and storage usage focus primarily on techniques that utilize various video encoder utilities to reduce bit rates. More processing power is used for more efficient encoding techniques. However, regardless of whether there is an event, the bits from the uploaded video are evenly distributed per second.

[0005] It would be desirable to implement intelligent time-lapse video with the aid of neural network input on a camera device to save bandwidth by reducing the bit rate of the video. Summary of the invention

[0006] The present invention relates to a device, which includes an interface and a processor. The interface can be configured to receive pixel data generated by a capture device. The processor can be configured to perform the following operations: receiving pixel data from the interface; generating a video frame in response to the pixel data; performing computer vision operations on the video frame to detect an object; performing classification of the detected object based on the characteristics of the object; determining whether the classification of the object corresponds to a user-defined event; generating a complete video stream in response to all video frames; and generating an encoded video frame according to the video frame. The complete video stream can be recorded to a storage medium local to the device. The encoded video frame can be transmitted to a cloud storage service. The encoded video frame can include a first sample of the video frame and a second sample of the video frame, the first sample being selected at a first rate when the user-defined event is not detected, and the second sample being selected at a second rate when the user-defined event is detected. The second rate can be greater than the first rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments of the present invention will become apparent from the following detailed description and appended claims and the accompanying drawings.

[0008] Figure 1 is a diagram illustrating an example context of the present invention.

[0009] Figure 2 is a diagram illustrating an example Internet-connected video camera implementing an example embodiment of the present invention.

[0010] Figure 3 is a block diagram illustrating components of an apparatus configured to provide event-centric time-lapse video with the aid of a neural network.

[0011] Figure 4 is a diagram showing interconnected cameras communicating with a cloud server and a video processing pipeline for generating time-lapse video.

[0012] Figure 5 is a diagram showing the smart interval shooting mode on the edge AI camera with CV analysis of all frames using events.

[0013] Figure 6 is a diagram showing recording of the full stream to a local storage medium and delivering a reduced bitrate video stream to a cloud storage service.

[0014] Figure 7 is a diagram illustrating event detection in captured video frames.

[0015] Figure 8 is a diagram showing an application for controlling preferences for time-lapse video operated on a smartphone. DETAILED DESCRIPTION

[0016] Embodiments of the present invention include providing smart time-lapse video on a camera device that saves bandwidth by reducing the bit rate of the video with the help of neural network input. Embodiments of the present invention can perform the following operations: (i) implement event detection on an edge device; (ii) implement video encoding on an edge device; (iii) generate smart time-lapse video with a varying frame rate based on detected events / objects; (iv) use a convolutional neural network implemented on a processor to detect objects / events; (v) adjust the time-lapse frame rate to capture all video frames when an event / object is detected; (vi) perform facial recognition and / or object classification; (vii) upload the encoded smart time-lapse video instead of a complete recording to a cloud storage server; (viii) implement configuration of parameters for uploading videos; (ix) reduce the amount of data transmitted to a cloud storage service; (x) reduce the amount of data stored using a cloud storage service; and / or (ix) be implemented as one or more integrated circuits.

[0017] Embodiments of the present invention may be configured to provide a smart interval shooting video mode. The interval shooting video mode may be a smart interval shooting video mode that automatically adjusts the video display speed based on the content captured in the video. Embodiments of the present invention may be configured to generate encoded smart interval shooting videos. Smart interval shooting videos may be generated on a cloud service (e.g., using scalable computing). Smart interval shooting videos may be generated on an edge device (e.g., an artificial intelligence (AI) camera).

[0018] Embodiments of the present invention may be configured to generate intelligent time-lapse videos with a video bit rate that is automatically adjusted based on the content. Video data with a bit rate adjusted based on the content may then be uploaded to a cloud service. In one example, the intelligent time-lapse videos may be used for security cameras. By adjusting the video bit rate of the intelligent time-lapse videos, the bit rate of the uploaded video stream may be significantly reduced. By adjusting the video bit rate of the intelligent time-lapse videos based on detected events, bandwidth costs and / or storage costs (e.g., locally or on a cloud service) may be reduced compared to uploading a complete video stream.

[0019] Edge AI cameras can be configured to implement artificial intelligence (AI) technology. Edge AI cameras use AI technology that can be more powerful in many ways than using cloud servers (e.g., by providing relevant data to users) and is a more energy-efficient solution. Edge AI cameras can be configured to execute computer-readable instructions on the device (e.g., without relying on external processing resources) to analyze video content frame by frame. Based on the analysis, the content can be tagged with metadata information. Metadata can be used to select video frames for time-lapse video. For example, video frames can be classified by being marked as not having an object / event of interest and having an object / event of interest. Computer vision (CV) operations can determine whether there is an object / event of interest.

[0020] When there is no CV event (or type of object) of interest for a video frame for a duration of N seconds (N=60 / 120 / 240 / …), the edge AI camera may select one of the video frames from the no-event duration. The selected frame may be used for video encoding (e.g., a video encoder built into the edge AI camera device). Selecting a video frame from the no-event duration N for encoding may be repeated for each N-second duration in which no event is detected. Selecting a video frame from the no-event duration N for encoding may result in an encoded output that provides a compressed portion of the captured video (e.g., thereby efficiently fast-forwarding through a “meaningless content” portion of the captured video at a high display speed).

[0021] When there is a detected CV event (or type of object) of interest for a video frame within a duration of M seconds (M=5 / 15 / 30 / …), the edge AI camera can adjust the number and / or selection rate of video frames within the event duration M. In an example, the object and / or event of interest can be a person detected, a car detected, an animal detected, an action, a specific face detected, etc. The selection of video frames can be adjusted to select all video frames from the M-second duration of the event (e.g., for a 2-minute event captured at 60 frames per second, all 7200 frames can be selected). The selection of video frames can be adjusted to select video frames at a higher rate than in the event-free duration within the M-second duration of the event (e.g., more frames are selected, but not all frames), for example, for a 2-minute event captured at 60 frames per second, the selected rate can be changed to 30 frames, and every other frame can be selected, resulting in the selection of 3600 video frames. The selected video frames can be encoded (e.g., using on-device video encoding of the AI ​​edge camera). Selecting video frames for encoding at a higher rate within the event duration M can result in an encoded output that provides a portion of the smart time-lapse video having a normal display speed for "meaningful content", or a portion of the smart time-lapse video having a slightly compressed display speed for "meaningful content" (but not as high as the speed for "meaningless content").

[0022] With the smart timelapse video mode implemented on the edge camera, a user can quickly browse video content captured over a long period of time (e.g., days / weeks / months), and the user can be confident that CV events of interest will not be missed. CV events of interest may include detecting people (e.g., known people) through face detection and / or face recognition, detecting car license plates (e.g., using a license plate reader), detecting pets (e.g., known animals) using animal / pet recognition, detecting motion (e.g., any motion detected above a predefined threshold), etc. The types of detected events and / or objects that may be considered events of interest may vary according to the design criteria of a particular implementation.

[0023] Embodiments of the present invention may enable a user to specify the types of objects / events that may be considered to be of interest. In one example, an application operating on a smartphone may be configured to adjust settings for an edge AI camera. In another example, the edge AI camera may be configured to provide a web interface to enable a user to remotely select objects / events that are considered to be events of interest. In yet another example, the edge AI camera may be connected to a cloud server, and the user may use a web interface to adjust settings stored on the cloud server, which may then be sent to the edge AI camera to control the types of timelapse objects of interest.

[0024] The duration of the timelapse can be configured as a regular interval, and can also be configured as the time since the last time the user has received a timelapse video. For example, if a user has missed 20 event notifications, at the moment the user interacts with the application (e.g., swiping on a smartphone) to view the event, a timelapse focused on the event can be presented to the user for viewing. Although embodiments of the present invention can generally be performed locally on an edge AI camera (e.g., thereby implementing a processor configured to implement a convolutional neural network and / or video encoding), embodiments of the present invention can be performed using software on a cloud server to achieve a similar effect.

[0025] Embodiments of the present invention may be configured to record full-time video content. Full-time video content may be stored to a local storage medium (e.g., eMMC, SD card, microSD card, etc.) on the edge AI camera. For example, constant video uploads may not be performed for all recorded content. For example, constant full video uploads may occupy upload bandwidth. Uploaded full videos may also result in more storage used in a cloud storage service (e.g., using more storage capacity in a cloud storage service may result in higher costs).

[0026] In some embodiments, instead of uploading a full video stream, a smart time-lapse video with a bit rate adjusted based on the content may be uploaded. Uploading the smart time-lapse video may result in both bandwidth cost savings and cloud storage cost savings. The smart time-lapse video may be presented to the user so that the user can see all detected events and a compressed version of the recorded video when no events are detected.

[0027] In one example, the smart time-lapse video may be presented to the user at a predetermined time (e.g., at the end of the day). In another example, the smart time-lapse video may be presented to the user at predetermined time intervals (e.g., every several hours / days / weeks, etc.). In yet another example, the smart time-lapse video may be presented to the user when an event of interest is detected. In yet another example, the smart time-lapse video may be presented on demand (e.g., based on a user request).

[0028] In some embodiments, no video may be uploaded to the cloud storage service most of the time (e.g., assuming that no events are typically detected). Smart timelapse video may provide a compressed version of non-events that enables the user to see what happened up until the moment of interest to the user (e.g., an event). In some embodiments, one or more smart timelapse videos of each event that has been detected since the last time the user requested to view video data (e.g., since the last time of real-time viewing, or since the last event viewed) may be presented to the user. Providing a smart timelapse video stream since the last time the user viewed may prevent continuous notifications from being sent to the user (e.g., multiple notifications sent to a smartphone may be a potential source of annoyance).

[0029] By selecting a video frame from the N-second duration of a non-CV event for encoding, the bit rate of the uploaded content can be reduced. Techniques for further reducing the bit rate can be used to encode the smart time-lapse video stream. The bit rate and / or amount of the uploaded content can be significantly reduced starting from the "no CV event" duration. The amount of data uploaded from the edge AI camera to the cloud storage server can be reduced. The amount of data transmitted from the cloud server to the end user (e.g., downloaded to a smartphone, desktop computer, tablet computing device, etc.) can also be reduced compared to the complete amount of data recorded (e.g., benefits can be achieved for both uploading and downloading). Users can quickly browse video content within a few days / weeks / months, and do not have to worry about whether the user may miss any CV events of interest.

[0030] refer to Figure 1 , a diagram illustrating an example context of the present invention is shown. A residence 50 and a vehicle 52 are shown. A camera system 100a-100n is shown. Each of the cameras 100a-100n can be configured to generate smart time-lapse videos. For example, each of the cameras 100a-100n can be configured to operate independently of each other. Each of the cameras 100a-100n can capture video and generate smart time-lapse videos. In one example, the corresponding smart time-lapse videos can be uploaded to a cloud storage service. In another example, the corresponding smart time-lapse videos can be stored locally (e.g., stored on a microSD card, stored to a local network attached storage device, etc.).

[0031] Each of the cameras 100a-100n can be configured to detect different or the same events / objects that can be considered to be of interest. For example, the camera system 100b can capture an area near the entrance of the residence 50. For the entrance of the residence 50, the object / event of interest can be detecting a person. The camera system 100b can be configured to analyze the video frames to detect a person, and when a person is detected, the smart interval shooting video can be slowed down (e.g., selecting a video frame for encoding at a higher frame rate). In another example, the camera system 100d can capture an area near the vehicle 52. For the vehicle 52, the object / event of interest can be detecting other vehicles and pedestrians. The camera system 100b can be configured to analyze the video frames to detect vehicles (or road signs) and people, and when a vehicle or pedestrian is detected, the smart interval shooting video can be slowed down.

[0032] refer to Figure 2 , a diagram illustrating an example Internet-connected camera implementing an example embodiment of the present invention is shown. Camera systems 100a-100n are shown. Each camera device 100a-100n can have a different style and / or use case. For example, camera 100a can be an action camera, camera 100b can be a ceiling-mounted security camera, camera 100n can be a network camera, etc. Other types of cameras (e.g., residential security cameras, battery-powered cameras, doorbell cameras, etc.) can be implemented. The design / style of cameras 100a-100n can vary depending on the design criteria of a particular implementation.

[0033] The camera systems 100a-100n may include a processor 102 and a capture device 104. The processor 102 may be configured to implement a convolutional neural network (CNN). The processor 102 may be configured to implement a video encoder. The processor 102 may generate smart time-lapse video. The capture device 104 may be configured to capture pixel data that may be used to generate video frames.

[0034] refer to Figure 3 , a block diagram illustrating components of an apparatus configured to provide event-centric time-lapse video with the aid of a neural network is shown.

[0035] The processor 102 may be configured to generate encoded video frames based on pixel data received from the capture devices 104a-104n. The processor 102 may be configured to select video frames to be encoded for smart time-lapse video. Video frames may be selected in response to objects / events detected by the CNN module 190b. The scheduler circuit 190a may load a directed acyclic graph (DAG) including parameters for detecting various types of objects / events. For example, different DAGs may be loaded in response to the type of event of interest selected by the user.

[0036] refer to Figure 4 , a diagram illustrating interconnected cameras communicating with a cloud server and a video processing pipeline for generating time-lapse video is shown. The edge AI camera can be configured to communicate with a remote cloud service and / or a user device (e.g., a smartphone). The user can select settings using an application on the smartphone 52b-52n. Preferences can be stored in the cloud service as event settings. The cloud service can provide neural network parameters (e.g., feature sets) to the processor 102. The processor 102 can receive feature sets to detect objects / events. The edge AI camera can be configured to upload encoded video frames (e.g., smart time-lapse videos) to the cloud service. The encoded video frames can be stored in a large-capacity video storage device of the cloud service. The user can use a smartphone to access the encoded video frames. The smartphone can connect to the large-capacity video storage device and download the smart time-lapse video. The cloud service can further provide notifications to the smartphone based on the content in the metadata tag of the smart time-lapse video.

[0037] The video processing pipeline of the processor 102 is shown. The processor 102 can generate raw video frames in response to pixel data captured by the capture device 104. The raw video frames can be analyzed by the CNN module. The CNN module can include an AI model (e.g., DAG). The CNN module can detect objects of interest based on the user's preferences (e.g., detecting one or all of a person, a vehicle, an animal, movement, a sound, etc. as an event).

[0038] When an event is detected, the CNN module can mark the video frame. The video frame marked with the event can be provided to the video encoding pipeline. The video frame with the detected event can be encoded at an adjusted frame rate (e.g., the full video or a higher frame rate than when the event is not detected).

[0039] The CNN module may determine which frames do not have detected events (e.g., video frames that are not of interest). Among the video frames that do not have events, the processor 102 may select a video frame for each duration of length N. For example, the processor 102 may select a video frame from the duration N in which the video is captured at 60 fps (e.g., select a frame from 60 frames captured in one second).

[0040] The unselected video frames may not be included in the smart time-lapse video stream. In one example, the unselected video frames may be discarded (e.g., not stored). In another example, the unselected video frames may be used in a recording (e.g., a recording of the complete video stream that may be stored locally on the camera device 100, stored on a microSD card, etc.). The selected video frames may be presented to a video encoding pipeline for encoding into a smart time-lapse video.

[0041] The video encoding pipeline can be configured to encode video frames in real time (e.g., HEVC, AVC, H264, etc.). The video encoding pipeline can insert selected video frames into the encoded video in sequence based on the time of capture. For example, each video frame can have a timestamp, and the video frames can be inserted into the encoded video in the order of the timestamps (e.g., the earliest video frame first to create a chronologically ordered video). The video encoding pipeline can generate an encoded video with video frames with events (displayed at normal speed) and video frames of non-events (displayed at a faster speed because fewer frames are selected).

[0042] The video encoding pipeline can be configured to apply various effects to selected video frames. For example, the video encoding pipeline can be configured to crop selected video frames, dewarp selected video frames, apply colorization (or remove color) based on selected video frames, apply blur to segments of video frames, etc. In an example, blur effects can be applied to portions of selected video frames to hide and / or censor specific objects (e.g., to hide the identity of known people for privacy reasons, to hide nude parts to prevent nude content from being uploaded, etc.).

[0043] Processor 102 may be configured to perform computer vision operations before video frames are encoded. Performing computer vision operations before video frames are encoded may enable computer vision operations to be performed without the cycle cost of first decoding the video frames. For example, if computer vision is performed by a cloud service, the encoded video frames are sent to a cloud server, and the cloud server wastes CPU cycles and power by first decoding the video frames, then performing computer vision operations, and then transcoding the time-lapse video.

[0044] The processor 102 may be configured to perform video frame selection for smart time-lapse video all locally at the edge AI camera 100. The processor 102 may implement a CNN module. The CNN module may be a hardware component (or multiple hardware components working together) of the processor 102 that implements a CNN-based detection technique (or other types of detection techniques) that all operate on the processor 102 to perform computer vision operations such as object detection, object tracking, object classification, etc. The hardware-accelerated CNN module may enable object / event detection to be performed on the camera system 100 (e.g., without relying on the computing resources of a cloud server to perform object detection). In some embodiments, the processor may use software acceleration for computer vision and / or a combination of hardware-accelerated computer vision and software-accelerated computer vision. Using computer vision operations running on a hardware-based CNN engine and / or a software-based CNN engine, the processor 102 implemented in the edge device 100 may be able to detect one or more types of events and apply event markers to captured video frames.

[0045] refer to Figure 5 , a diagram illustrating a smart interval shooting mode on an edge AI camera with CV analysis of all frames using events is shown. A captured video stream including a stream of video frames (e.g., a group of pictures (GOP)) is shown. The CNN module can perform computer vision to determine which video frames include CV events. The first four groups of duration N of video frames may not include CV events. One of the video frames from each of the first four groups of duration N can be selected by the processor 102 for smart interval shooting video (e.g., four non-event video frames are selected).

[0046] After four non-event groups, a CV event may be detected (e.g., of duration M). For a complete smart time-lapse video, all video frames with CV events within duration M may be selected for the smart time-lapse video stream. In the example shown, there may be six video frames within duration M. All six video frames with CV events may be selected for the smart time-lapse video.

[0047] After the duration M of the CV event, two other groups of duration N of video frames that do not include the CV event may be detected. The frame rate for video selection may be readjusted back to the default rate. For example, a video frame may be selected for each of the N groups for the duration of the next two non-events of the video frame. Although the first video frame is shown as being selected from the non-CV event group of video frames, any one of the video frames may be selected from each of the non-CV event groups according to the design criteria of a particular implementation.

[0048] Smart interval shooting video can be played back at a normal frame rate. Since there are fewer video frames to be played back from the non-CV portion of the video frame, the playback of the non-CV portion of the smart interval shooting video stream may appear to be played back at a higher speed than real-time playback. For example, if 60 non-CV event groups of video frames have been detected, a frame can be selected from each frame for the smart interval shooting video. Playing back the smart interval shooting video at 60fps can cause the non-CV event group to be played back in a compressed form of one second. Since all the video frames of the CV event have been selected, the playback of the portion of the smart interval shooting video stream can be played back at a normal speed (e.g., in real time).

[0049] The processor 102 can generate a smart time-lapse video based on the detected event. The CNN module can detect events and mark video frames corresponding to specific events. In one example, a video frame (e.g., frame N) can correspond to event A and event B, and the next one hundred video frames after frame N (e.g., frame N+100) can correspond only to event B. The processor 102 can be configured to combine the selected frames in the following manner: select some video frames from a video without an event (e.g., typically only select I-frames), and then select all frames with event A markings. Then, the encoding device can create a smart time-lapse video (e.g., compressed video) including event A (e.g., by finding frame N and any adjacent frames with event A markings that can be selected). Similarly, different smart time-lapse videos (e.g., separate video files) can be generated for event B (e.g., by finding video frame N and the next frame N+100 with event B markings).

[0050] refer to Figure 6 , a diagram illustrating recording a complete stream to a local storage medium and delivering a reduced bit rate video stream to a cloud storage service is shown. A captured video stream is shown. The captured video stream may be all of the video data (or content) captured by the camera system 100. The camera system 100 may record all of the captured video streams (e.g., the complete recording) to a local storage medium. In the example shown, the local storage medium may be a microSD card. In another example, the local storage medium may be a hard drive and / or a solid state drive. In yet another example, the local storage medium may be an eMMC. The type of local storage medium implemented may vary depending on the design criteria of a particular implementation.

[0051] Smart time-lapse video is shown. The smart time-lapse video can have an adjusted bit rate (e.g., a compressed version of the recorded video corresponding to non-CV events and a complete version of the recorded video corresponding to CV events). In some embodiments, the CV event data can also be a compressed version (e.g., using partial frames of an event that still provides selected video frames (e.g., more detailed, slower playback speed), rather than a compressed view of selected non-CV event video frames). The smart time-lapse video is shown as being transmitted to a cloud storage service. For example, video frames can be selected, encoded into a smart time-lapse video stream, and then uploaded to a cloud storage service.

[0052] The complete record of the captured video stream stored to the local storage medium is shown to include more video frames than the smart time-lapse video stream. Since the smart time-lapse video stream includes fewer video frames, the smart time-lapse video stream can have a lower bit rate than the complete record. Uploading the smart time-lapse video stream can transmit less video data than uploading the complete record (e.g., consume less bandwidth, require less storage space in the cloud service for storage, etc.). The smart time-lapse video stream can provide the same amount of detail of the CV event as the complete record (or if the CV event data has a slightly reduced frame rate, the smart time-lapse video stream can provide enough detail), and provide a compressed view of non-CV events. Uploading the smart time-lapse video stream can generally provide the user with the same amount of detail of the CV event as the complete record, but consume less bandwidth and / or use less storage space.

[0053] The benefits of uploading a smart time-lapse video stream (e.g., less bandwidth) can also be realized when downloading a smart time-lapse video stream from a cloud storage service. For example, a user can request to view a smart time-lapse video stream using a smartphone app. For example, if the data of a complete recording is twice that of a smart time-lapse video stream, there can be a four-fold savings in bandwidth for both the upload and subsequent download (e.g., two times the savings for uploads and another two times the savings for downloads).

[0054] In some embodiments, the complete video stream can be uploaded to a cloud storage service, and the distributed processing of the cloud storage service can be operated to select video frames corresponding to CV events. Although uploading the complete video stream to the cloud storage service may not provide bandwidth savings, storage savings can still be achieved (e.g., after generation by distributed processing, the complete video record can be replaced by the smart time-lapse video stream). In addition, bandwidth savings can be achieved for subsequent downloads (e.g., a user downloads the smart time-lapse video stream from the cloud storage service).

[0055] In some embodiments, video frames can be uploaded at a full frame rate (or fixed frame rate) only when an event is detected by the processor. When no event is detected, video frames can be uploaded at very large intervals to save the amount of bits to be uploaded. In one example, the camera system 100 can choose to store the video locally (e.g., SD card, eMMC, etc.) and only upload the smart interval shooting video. In the example, the smart interval shooting video can be uploaded at the end of the day, or when the user opens the application to see what has happened since the last time the user has viewed the application. In another example, the camera system 100 can still upload / encode the 24×7 stream to the cloud storage service. At the end of the day, a push notification (or email) can be sent to the end user. In contrast to entering the timeline view and viewing the event video clips (which will result in data consumption and cloud costs), the smart interval shooting video can be presented to the user. The user can see important events, and the camera operator saves costs (e.g., saving the cost of scrolling and checking each of the individual clips).

[0056] exist Figure 5 and Figure 6 , the video frames selected by the processor 102 to be encoded for the smart time-lapse video are shown as larger than the frames that may not be selected. For illustrative purposes, the sizes of the selected video frames and / or the unselected video frames are shown as different (e.g., a larger size indicates that the video frame has been selected, and a smaller size indicates that the video frame has not been selected). In general, the selected video frames may not necessarily be video frames of a particular type (e.g., the selected video frames may be any of an I-frame, a B-frame, a P-frame, etc.). For example, all video frames in the "CV event duration" may be part of the same coded GOP as the video frames in the "no CV event duration", which may enable the selected video frames to be any type of video frames. In the example, the selected frames may not all need to be I-frames, because when the video frames are captured and the processor 102 is performing encoding, object / event detection may be performed in parallel by the CNN module (e.g., the computer vision operations by the CNN module may not have to be completed as soon as a new video frame is captured). Computer vision operations may be performed, and then if an event is detected, the entire video GOP may be marked as an event (e.g., event A) without changing the video GOP for the event. Processor 102 can decode the entire GOP and select some of the frames using the CNN module. For example, for a portion of the smart time-lapse video (in Figure 6 ), for better quality, the CNN module may decide to search only some of the video frames in a GOP.

[0057] refer to Figure 7, a diagram illustrating event detection in a captured video frame is shown. An example video frame is shown. The example video frame may include pixel data provided to the CNN module. The CNN module may be configured to analyze the pixel data of the video frame to determine whether an event / object of interest exists. The type of object / event detected by the CNN module may be defined by a feature set selected by a user and / or loaded by the processor 102.

[0058] An example video frame may be a captured video of the front of residence 50 (e.g., a view from an angle looking down the driveway from the front door). An example video frame may include a person signing for a package delivered by a delivery person. A delivery truck may be parked on the road. Trees and nearby residences are also shown.

[0059] A dashed box representing a bounding box of an object detected by the CNN module is shown. The detected objects may be a face of a delivery person, a face of a person (e.g., a homeowner), and a delivery vehicle. In an example, the settings for processor 102 may define objects of interest as people and vehicles. For example, trees and homes may not be objects of interest.

[0060] Since the example video frame includes at least one object of interest, the CNN module can mark the video frame as an "event" video frame. If the smart time-lapse video includes all video frames within the duration of the event, the example video frame can be encoded and transcoded as a smart time-lapse video frame.

[0061] refer to Figure 8 , a diagram illustrating an application operating on a smartphone for controlling preferences for timelapse video is shown. An application interface is shown. The smartphone can be configured to connect directly to the edge AI device. The smartphone can be configured to connect to a cloud service to access user preferences stored in the cloud. The edge AI device can be configured to operate according to settings retrieved from the cloud.

[0062] The application can allow the user to choose whether to encode the complete smart interval shooting video stream (e.g., select all video frames with detected events / objects) or to encode a portion of the interval shooting video stream. If the partial smart interval shooting video stream option is selected, the user can further define the selected frame rate for the video frames with detected events. In the example shown, 60fps, 30fps, and 15fps are shown as available options. The available options for the frame rate of the partial smart interval shooting video can vary according to the design criteria of the specific implementation. Selecting a lower frame rate can result in a more compressed version of the detected event being inserted into the smart interval shooting video stream.

[0063] The application can enable the user to select the type of objects and / or events to be detected. For example, an event can be the detection of a person, a vehicle, an animal (e.g., using computer vision), detecting an action, etc. An event can be an audio event (e.g., detecting sounds that may be louder than a predefined sound level, a specific type of sound signature, etc.). The application can further enable the user to watch a live stream (e.g., displaying video data currently being captured by the edge AI device). The application can provide a notification when an event is detected by the CNN module.

[0064] The application can enable the user to select when the smart time-lapse video is delivered to the user (e.g., when it is pushed from the cloud storage service to the user's smartphone). The smart time-lapse video can be delivered at predetermined time intervals. In the example shown, the predetermined time interval can be hourly, daily, or weekly. For example, if the user selects the daily option, the cloud storage service can deliver the smart time-lapse video (e.g., encoded video frames) every day. For example, other options can be provided to select the time of day to deliver (e.g., midnight every day). Providing smart time-lapse videos every day can enable users to quickly watch videos of events of interest during a specific time frame (e.g., the user will not miss events that have occurred since the previous video was viewed).

[0065] In some embodiments, a user may choose to receive smart time-lapse video on demand. For example, providing smart time-lapse video on demand may include transmitting a smart time-lapse video each time an event has been detected. Smart time-lapse videos may have various lengths (e.g., depending on how much time has passed between events, they may be a few hours apart or a few days apart). When an event is detected, a smart time-lapse video may be transmitted to the user's smartphone, which includes all encoded video frames since the previous event was detected. Providing all encoded video frames since the previous event may enable the user to watch each event shortly after the event occurs. Providing all encoded video frames since the previous event may enable the user to view a compressed version of the video frames until before the event, and then watch the full frame rate video of the actual event.

[0066] The terms "may" and "generally" when used in conjunction with "is" and a verb herein are meant to convey such an intention: the description is exemplary and is considered broad enough to cover both the specific examples presented in this disclosure and alternative examples that can be derived based on this disclosure. The terms "may" and "generally" as used herein should not be interpreted as necessarily implying the desirability or possibility of omitting the corresponding element.

[0067] While the invention has been particularly shown and described with reference to embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the scope of the invention.

Claims

1. A device comprising: an interface configured to receive pixel data generated by a capture device; as well as a processor configured to: (i) receive the pixel data from the interface; (ii) generate video frames in response to the pixel data; (iii) perform computer vision operations on the video frames to detect objects; (iv) perform classification of the detected objects based on characteristics of the objects to generate metadata for each of the video frames; (v) determining whether the classification of the object corresponds to a user-defined event; (vi) recording a complete video stream to a storage medium local to the device in response to all of the video frames; and (vii) generating an encoded video frame based on the video frame, wherein (a) the metadata enables: (i) marking the video frame as an event video frame if the classification of the object corresponds to the user-defined event; and (ii) marking the video frame as a non-event video frame if the classification of the object does not correspond to the user-defined event; (b) the encoded video frames are transmitted to a cloud storage service; (c) the encoded video frames include: (i) a plurality of groups of pictures including the non-event video frames; and (ii) at least one group of pictures including the event video frames; and (d) The encoded video frame is generated in response to: (i) selecting a video frame from each of the plurality of picture groups that include the non-event video frames; and (ii) selecting more than one of the event video frames from the picture groups that include the event video frames.

2. The device according to claim 1, wherein: The encoded video frames provide a time-lapse video stream.

3. The device according to claim 1, wherein: The encoded video frames are uploaded to the cloud storage service at a specific time.

4. The device according to claim 3, wherein: The specific time includes a user-selected time of day.

5. The device according to claim 3, wherein: The specific time includes a predefined interval.

6. The device according to claim 3, wherein: The specific time is selected in response to detecting the user-defined event.

7. The device according to claim 3, wherein: The specific time is the time at which the user requests to view the encoded video frame.

8. The device according to claim 3, wherein: Video data is not transmitted between the device and the cloud storage service until the specific time.

9. The device according to claim 1, wherein: The complete video stream has a larger data size than the encoded video frame.

10. The device according to claim 1, wherein: Uploading the encoded video frames to the cloud storage service instead of uploading the complete video stream reduces: (a) the amount of bandwidth used for transmission between the device and the cloud storage service; and (b) an amount of storage capacity used by the cloud storage service.

11. The device according to claim 1, wherein: A user uses a smartphone application to access the encoded video frames stored on the cloud storage service.

12. The device according to claim 11, wherein: (i) the smartphone application stores a timestamp corresponding to the last time the user viewed the encoded video frame; and (ii) presenting to the user the encoded video frames stored by the cloud storage service since the last time the user viewed the encoded video frames.

13. The device according to claim 1, wherein: The storage medium local to the apparatus comprises a microSD flash memory device.

14. The device according to claim 1, wherein: All of the event video frames are selected from the group of pictures including the event video frames to provide a frame rate that is the same as a frame rate of the complete video stream for the portion of the encoded video frames including the event video frames.

15. The device according to claim 1, wherein: Less than all of the event video frames are selected from the group of pictures that includes the event video frames to provide a frame rate for the portion of the encoded video frames that includes the event video frames that is lower than a frame rate of the complete video stream and higher than a frame rate of the portion of the encoded video frames that includes the non-event video frames.

16. The device according to claim 1, wherein The computer vision operations are performed on an integrated circuit.

17. The device according to claim 1, wherein: (i) the apparatus and the capture device are implemented on an edge device; and (ii) the edge device communicates with the cloud storage service.

18. The device according to claim 1, wherein: The selected video frame includes any one of an I-frame, a B-frame or a P-frame.

19. The device according to claim 1, wherein: (i) the complete video stream is uploaded to the cloud storage service; and (ii) the encoded video frames are transmitted from the cloud storage service to the user (a) in response to a request to view video data or (b) at predetermined time intervals.

Images