Camera time-based motion trajectory and motion heat map for periodically captured images

Abstract

Disclosed herein are embodiments of systems, methods, and / or computer program products, and combinations and sub-combinations thereof, for generating long exposure composite image (LECI) frames that provide a summary to monitor activity in an environment over a duration. One embodiment uses an image capture device that captures a sequence of image frames over a duration of time using an optical sensor of the image capture device. Embodiments then generate the LECI frames by combining the sequence of image frames using an embedded low power processor coupled to the optical sensor and dedicated to generating the LECI frames. Embodiments then transmit the LECI frame to the network device using a transceiver coupled to the low power processor.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Non-Provisional Application No. 18 / 486,464, filed on October 13, 2023, entitled “CAMERA TIME-BASED MOTION TRAJECTORY AND MOTION HEATMAPS FOR PERIODICALLY CAPTURED IMAGES,” the entire contents of which are incorporated herein by reference. Background Art Technical Field

[0003] The present disclosure relates generally to image processing, and more particularly to generating a composite image that represents a visual summary of a captured image and that can include a captured image-based heat map and a camera-time-based motion trajectory. Summary of the invention

[0004] Provided herein are embodiments of systems, apparatus, articles of manufacture, methods, and / or computer program products for generating long exposure synthetic image (LECI) frames that provide a single-image visual summary of activity in a monitored environment over a duration of time, and / or combinations and sub-combinations thereof.

[0005] Some aspects of the present disclosure relate to a method for generating a LECI frame. According to some aspects, the method includes capturing a sequence of image frames of a monitored environment over a duration using an optical sensor of an image capture device. In some aspects, this step can be performed by causing the optical sensor to always record the monitored environment. The method also includes generating the LECI frame by combining the sequence of image frames using an embedded low-power processor coupled to the optical sensor and dedicated to generating LECI frames. The embedded low-power processor can be configured to run separately from the main processor of the image capture device, and can even be configured to run when the main processor is in a sleep or low-power mode. The generated LECI frame can then be transmitted to a network device using a transceiver coupled to the embedded low-power processor.

[0006] According to some aspects, image frames within a sequence of image frames can be weighted based on a time period during which the image frames were captured. For example, the sequence of image frames includes a first group of image frames and a second group of image frames, wherein the first group of image frames is associated with a first time period within a duration, and the second group of image frames is associated with a different time period within the duration. The first time period is closer than the second time period, and generating a composite image frame can include performing a weighted average of the image frames within the sequence. The weighted average can include, for example, assigning a first set of weights to a first group of image frames of the sequence of image frames, and assigning a second set of weights to a second group of image frames, wherein the first set of weights is greater than the second set of weights.

[0007] According to some aspects, the method also includes generating a LECI frame by combining a sequence of image frames by calculating a pixel-by-pixel weighted average of pixel values ​​on the sequence of image frames based on a first set of weights and a second set of weights. According to some aspects, the method also includes generating a LECI by performing image detection on one or more image frames. Image detection can, for example, be used to identify moving objects in one or more image frames of the sequence of image frames, and weighted averaging is performed by assigning higher weights to pixels corresponding to the moving objects in the one or more image frames. According to some aspects, the method also includes generating a LECI frame by identifying a relative displacement of one or more pixels between a set of image frames of the sequence of image frames to generate one or more annotations in the LECI frame. Examples of some annotations include a direction vector (e.g., an arrow) indicating a direction of motion of an object in the sequence of image frames and a heat map indicating the length of time an object stays at a specific location in a monitored environment.

[0008] According to some aspects, the method further comprises capturing the sequence of image frames at a capture rate proportional to the activity metric in the monitored environment. According to some aspects, the duration is inversely proportional to the activity metric in the monitored environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings are incorporated herein and constitute a part of this specification.

[0010] Figure 1 A block diagram of a multimedia environment is shown in accordance with some embodiments.

[0011] Figure 2 A block diagram of a streaming media device according to some embodiments is shown.

[0012] Figure 3 A block diagram of an example image capture device for generating long exposure composite image (LECI) frames is shown in accordance with some embodiments.

[0013] Figure 4A An exemplary LECI frame according to some embodiments is shown.

[0014] Figure 4B Another exemplary LECI frame is shown in accordance with some embodiments.

[0015] Figure 5 A flow chart of a method for generating LECI frames from stored video data according to some embodiments is shown.

[0016] Figure 6 A flow chart of a method for generating LECI frames from a sequence of image frames according to some embodiments is shown.

[0017] Figure 7A flow chart of a method for identifying movement of large objects between frames of video data is shown in accordance with some embodiments.

[0018] Figure 8 An example computer system is shown for implementing various embodiments.

[0019] In the drawings, like reference numbers generally refer to like or similar elements. Also, generally, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. DETAILED DESCRIPTION

[0020] Embodiments of systems, apparatus, articles of manufacture, methods, and / or computer program products for generating long exposure composite image (LECI) frames, and / or combinations and sub-combinations thereof, are provided herein, wherein the LECI provides a single image visual summary of an activity occurring over a duration. For example, aspects of the present invention describe capturing a sequence of image frames at a low frame rate, and generating a LECI frame by combining the sequence of image frames.

[0021] Surveillance cameras capture and process images and videos for long periods of time on limited battery power. Maximizing the battery life of such surveillance cameras is critical to their effectiveness and longevity. Cameras operating at low frame rates generally consume less power and generate less data, thereby reducing storage costs. However, lower frame rates can result in more shaky video, especially for fast-moving objects. Therefore, the frame rate of a surveillance camera involves a trade-off between capturing smooth video and conserving power and storage resources.

[0022] Additionally, reviewing surveillance camera recorded video over long periods of time can be challenging. Using tools such as increased playback speeds, time-stamped logs, and motion detection alerts may not be the most efficient method of video review (especially when the goal is to quickly identify relevant events while minimizing time spent on irrelevant footage).

[0023] Aspects of the present disclosure describe an efficient method for summarizing activities occurring in a monitored environment over a duration. According to some aspects, a LECI frame is generated using a sequence of image frames captured over the duration. Each LECI frame summarizes activities occurring in the monitored environment over the duration using a single image, thereby reducing bandwidth and storage requirements. According to some aspects, a LECI frame is generated by combining a sequence of image frames by calculating a pixel-by-pixel weighted average of pixel values ​​over the sequence of image frames.

[0024] Various aspects of the present disclosure can be used Figure 1 The multimedia environment 102 shown may be implemented and / or may be Figure 1102. However, it should be noted that multimedia environment 102 is for illustration purposes only and is not limiting. As persons skilled in the relevant art will appreciate based on the teachings contained herein, embodiments of the present disclosure can be implemented using and / or can be a part of an environment other than and / or in addition to multimedia environment 102. An example of multimedia environment 102 will now be described.

[0025] Multimedia Environment

[0026] Figure 1 A block diagram of a multimedia environment 102 according to some embodiments is shown. In a non-limiting example, the multimedia environment 102 can involve streaming media. However, the present disclosure is applicable to any type of media (instead of or in addition to streaming media) and any mechanism, device, protocol, method and / or process for distributing media.

[0027] The multimedia environment 102 can include one or more media systems 104. The media system 104 can represent a family room, a kitchen, a backyard, a home theater, a school classroom, a library, a car, a boat, a bus, an airplane, a movie theater, a stadium, an auditorium, a park, a bar, a restaurant, or any location or space where it is desirable to receive and play streaming content. One or more users 132 can operate with the media system 104 to select and consume content.

[0028] Each media system 104 can include one or more media devices 106, each coupled to one or more display devices 108. It should be noted that unless otherwise specified herein, terms such as "coupled," "connected," "attached," "linked," "combined," etc. can refer to physical, electrical, magnetic, logical, etc. connections.

[0029] To name a few examples, the media device 106 can be a streaming device, a DVD or Blu-ray device, an audio / video playback device, a set-top box, and / or a digital video recording device. To name a few examples, the display device 108 can be a monitor, a television (TV), a computer, a smart phone, a tablet, a wearable device (e.g., a watch or glasses), an appliance, an Internet of Things (IoT) device, and / or a projector. In some embodiments, the media device 106 can be part of its corresponding display device 108, integrated with its corresponding display device 108, operably coupled to its corresponding display device 108, and / or connected to its corresponding display device 108. In some embodiments, the image capture device 134 can be operably coupled to and / or connected to the media system 104 and communicate with one or more content servers 120 and / or one or more system servers 126 via the media system 104. In some aspects, the image capture device 134 can communicate directly with one or more content servers 120 and / or one or more system servers 126 without communicating via the media system 104.

[0030] Each media device 106 can be configured to communicate with a network 118 via a communication device 114. The communication device 114 can include, for example, a cable modem or a satellite television transmitter. The media device 106 can communicate with the communication device 114 via a link 116, which can include a wireless (e.g., Wi-Fi) connection and / or a wired connection.

[0031] In various embodiments, network 118 can include, but is not limited to, wired and / or wireless local area networks, extranets, the Internet, cellular networks, Bluetooth, infrared and / or any other short-range, long-range, local, regional or global communication mechanisms, devices, methods, protocols and / or networks, and any combination thereof.

[0032] The media system 104 can include a remote control 110. The remote control 110 can be any component, part, means and / or method for controlling the media device 106 and / or the display device 108, such as a remote control, a tablet computer, a laptop computer, a smart phone, a wearable device, an on-screen controller, an integrated control button, an audio controller, or any combination thereof, to name a few examples. In an embodiment, the remote control 110 communicates wirelessly with the media device 106 and / or the display device 108 using a cellular network, Bluetooth, infrared, etc., or a combination thereof. The remote control 110 can include a microphone 112, which will be further described below.

[0033] The multimedia environment 102 can include a plurality of content servers 120 (also referred to as content providers, content channels, or content sources). Figure 1Only one content server 120 is shown in the figure, but in fact the multimedia environment 102 can include any number of content servers 120. Each content server 120 can be configured to communicate with the network 118.

[0034] Each content server 120 can store content 122 and metadata 124. Content 122 can include any combination of music, video, movies, television programs, multimedia, images, still pictures, text, graphics, game applications, advertisements, program content, public service content, government content, local community content, software, and / or any other content or data object in electronic form.

[0035] In some embodiments, metadata 124 includes data about content 122. For example, metadata 124 can include associated or auxiliary information indicating or relating to writers, directors, producers, composers, artists, actors, summaries, chapters, productions, history, years, trailers, alternate versions, related content, applications, and / or any other information related to or associated with content 122. Metadata 124 may also or alternatively include links to any such information related to or associated with content 122. Metadata 124 may also or alternatively include one or more indexes to content 122.

[0036] The multimedia environment 102 can include one or more system servers 126. The system servers 126 can be run from the cloud to support the media devices 106. It should be noted that the structural and functional aspects of the system servers 126 can reside in whole or in part in the same or different ones of the system servers 126.

[0037] The media devices 106 can exist in thousands or millions of media systems 104. Thus, the media devices 106 can be suitable for crowdsourcing embodiments, and thus, the system servers 126 can include one or more crowdsourcing servers 128.

[0038] For example, using information received from thousands or millions of media devices 106 in media systems 104, one or more crowdsourcing servers 128 can identify similarities and overlaps between closed captioning requests issued by different users 132 watching a particular movie. Based on such information, one or more crowdsourcing servers 128 can determine that turning on closed captioning can enhance the user's viewing experience during specific parts of the movie (e.g., when the soundtrack of the movie is difficult to hear), and turning off closed captioning can improve the user's viewing experience during other parts of the movie (e.g., when displaying closed captioning obstructs key visual aspects of the movie). As a result, one or more crowdsourcing servers 128 can operate to automatically turn on and / or turn off closed captioning during future streaming of the movie.

[0039] The system server 126 can also include an audio command processing module 130. As described above, the remote control 110 can include a microphone 112. The microphone 112 can receive audio data from a user 132 (as well as other sources such as a display device 108). In some embodiments, the media device 106 can be audio-responsive, and the audio data can represent spoken commands from the user 132 to control the media device 106 and other components (e.g., the display device 108) in the media system 104.

[0040] In some embodiments, audio data received by microphone 112 in remote control 110 is transmitted to media device 106, which then forwards the audio data to audio command processing module 130 in system server 126. Audio command processing module 130 can operate to process and analyze the received audio data to identify spoken commands of user 132. Audio command processing module 130 can then forward the spoken commands back to media device 106 for processing.

[0041] In some embodiments, the audio data can be processed and analyzed by the audio command processing module 216 in the media device 106 (see Figure 2 ). The media device 106 and the system server 126 can then collaborate to select one of the spoken commands (whether a spoken command recognized by the audio command processing module 130 in the system server 126 or a spoken command recognized by the audio command processing module 216 in the media device 106) for processing.

[0042] Figure 2 A block diagram of an example media device 106 according to some embodiments is shown. The media device 106 can include a streaming module 202, a processing module 204, a memory / buffer 208, and a user interface module 206. The user interface module 206 can include an audio command processing module 216, as described above.

[0043] The media device 106 can also include one or more audio decoders 212 and one or more video decoders 214 .

[0044] Each audio decoder 212 can be configured to decode audio in one or more audio formats, such as but not limited to AAC, HE-AAC, AC3 (Dolby Digital Surround), EAC3 (Dolby Digital Surround Plus), WMA, WAV, PCM, MP3, OGG, GSM, FLAC, AU, AIFF and / or VOX, to name a few examples.

[0045] Likewise, each video decoder 214 can be configured to decode video in one or more video formats, such as, but not limited to, MP4 (mp4, m4a, m4v, f4v, f4a, m4b, m4r, f4b, mov), 3GP (3gp, 3gp2, 3g2, 3gpp, 3gpp2), OGG (ogg, oga, ogv, ogx), WMV (wmv, wma, asf), WEBM, FLV, AVI, QuickTime, HDV, MXF (OP1a, OP-Atom), MPEG-TS, MPEG-2PS, MPEG-2TS, WAV, Broadcast WAV, LXF, GXF and / or VOB, to name a few examples. Each video decoder 214 can include one or more video codecs, such as but not limited to H.263, H.264, H.265, AVI, HEV, MPEG1, MPEG2, MPEG-TS, MPEG-4, Theora, 3GP, DV, DVCPRO, DVCPRO, DVCProHD, IMX, XDCAM HD, XDCAM HD422 and / or XDCAM EX, to name a few examples.

[0046] The stream module 202 of the media device 106 can be configured to receive image information from the image capture device 134. In some aspects, the image information can include a LECI frame generated by a low power processor of the image capture device 134. In some aspects, the image information can include a sequence of image frames recorded by the image capture device 134 and an indication (e.g., a bit in a packet header, a flag) that the media device 106 can generate a LECI frame from the provided image information. For example, the processing module 204 can receive the sequence of image frames from the image capture device 134 and generate the LECI frame according to the provided sequence. In this manner, the image capture device 134 can offload LECI processing to the media device 106. For example, the image capture device 134 can determine that it lacks sufficient processing power or power (e.g., low battery) to generate the LECI frame, and instead transmits the recorded sequence of image frames to the media device 106.

[0047] Now refer to Figure 1 and Figure 2In some embodiments, the user 132 can interact with the media device 106 via, for example, the remote control 110. For example, the user 132 can interact with the user interface module 206 of the media device 106 using the remote control 110 to select a content item, such as a movie, a television program, music, a book, an application, or a game. In response to the user selection, the streaming module 202 of the media device 106 can request the selected content item from one or more content servers 120 via the network 118. The one or more content servers 120 can transmit the requested content item to the streaming module 202. The media device 106 can transmit the received content item to the display device 108 for playback to the user 132.

[0048] In some aspects, the media device 106 can display an interface for interacting with the sequence of image frames provided by the image capture device 134. For example, the interface can display selectable options for generating LECI frames based on the sequence of image frames. An example of a selectable option is the duration of the image sequence for generating LECI images (e.g., 1 minute, 5 minutes). Another example includes the type of annotation or effect (e.g., arrows, heat maps, highlights, blurs) to be added to the LECI to represent actions or objects detected within the frames of the sequence of frames.

[0049] In a streaming embodiment, the stream module 202 can transmit content items to the display device 108 in real time or near real time as such content items are received from one or more content servers 120. In a non-streaming embodiment, the media device 106 can store content items received from one or more content servers 120 in the memory / cache 208 for later playback on the display device 108.

[0050] Generate long exposure composite image frames

[0051] Figure 3 A block diagram of an example image capture device 300 for generating a long exposure composite image (LECI) frame according to some aspects of the present disclosure is shown. According to some aspects, the image capture device 300 (which can be Figure 1 The image capture device 134 can be configured to Figure 1 The image capture device 300 may be configured to communicate with a media system 104, one or more content servers 120, or one or more system servers 126 in the multimedia environment 102. According to some aspects, the image capture device 300 may be a security camera, a CCTV camera, a digital camera, a smartphone, a webcam, a video camera, a medical imaging device, and a thermal imaging camera, to name a few examples.

[0052] In the example shown, image capture device 300 includes optical sensor 302, image processor 306, video encoder 312, central processor 314, storage module 308, battery module 310, and transceiver. Optical sensor 302 can include an embedded low power processor 304 dedicated to generating LECI.

[0053] According to some aspects, the optical sensor 302 can be configured to capture images and / or video of the monitored environment. The optical sensor 302 can have two components: an array of photosensitive pixels that convert photons into electrical charge and a readout logic circuit that converts the charge into a digital value (i.e., a raw image pixel). According to some aspects, the optical sensor 302 can be a charge coupled device (CCD) sensor that uses a capacitor arrangement to move charge from one pixel to another until the charge is read from the sensor. Alternatively, the optical sensor 302 can be a complementary metal oxide semiconductor (CMOS) sensor that uses a transistor at each pixel location to amplify and read the charge.

[0054] According to some aspects, the optical sensor 302 generates uncompressed raw pixel data corresponding to the captured image and / or video. According to some aspects, the optical sensor 302 can capture color information by using a pattern of red, green, and blue (RGB) filters on the pixels. According to some aspects, the optical sensor 302 can capture the luminance (Y) and chrominance (UV) components of the pixels. The uncompressed raw pixel data generated by the optical sensor 302 can then be processed by the embedded low-power processor 304 and / or the image processor 306 of the image capture device 300.

[0055] According to some aspects, the embedded low-power processor 304 is a dedicated integrated circuit that is designed to provide processing capabilities for specific tasks (e.g., calculating a weighted average of a set of image frames) to reduce the power consumption of the image capture device 300. The embedded low-power processor 304 can minimize power consumption through a combination of architectural design and power management techniques. For example, the embedded low-power processor 304 can use an energy-saving microarchitecture to reduce power consumption. In addition, compared to the main processor of the image capture device 300 (i.e., the image processor 306), the embedded low-power processor 304 can run at a lower clock speed and voltage, and the slower clock speed and reduced voltage level result in lower power consumption.

[0056] According to some aspects, raw image and / or video pixels are transmitted from the optical sensor 302 (e.g., via a camera serial interface) to an image processor 306. The image processor 306 can perform image compression by generating quantized pixels. In addition, the image processor 306 can also perform noise reduction, color correction, gamma correction, and other enhancements to improve the overall quality of the image.

[0057] According to some aspects, the operations performed by the image processor 306 can consume a large amount of battery power. Therefore, the embedded low-power processor 304 can be configured to operate when the image capture device 300 is in a power state for saving power (e.g., a low-power or sleep state). The operation of the embedded low-power processor 304 (e.g., generating LECI frames) is configured to be substantially energy-efficient compared to operation using the main processor (i.e., the image processor 306). In order to reduce power consumption, the image processor 306 can implement a sleep cycle. The sleep cycle can be designed to reduce power consumption during inactivity while still allowing the sensor to quickly resume normal operation when needed.

[0058] According to some aspects, the video encoder 312 obtains raw video data from the optical sensor 302 and converts it into a digital format that can be stored or transmitted over a network. The video encoder 312 can be embedded in the system on chip (SoC) of the image capture device 300. According to some aspects, the video encoder 312 can use a video compression algorithm to reduce the size of the video data. Common video compression standards include H.264, H.265 (High Efficiency Video Coding (HEVC)) and VP9. According to some aspects, in order to reduce power consumption, instead of the central processor 314, the video encoder 312 can be configured to generate motion vectors from a sequence of image frames of the input video data. In other words, the video encoder 312 can be configured to perform frame-by-frame comparison when compressing the video. The video encoder 312 provides low-power hardware acceleration for the image capture device 300 because the generation of motion vectors is offloaded to the video encoder 312. Motion vectors can be used to describe how objects in a video move from one frame to another. The output of the video encoder 312 can be a compressed video bitstream. The central processor 314 can scan the generated bitstream for motion vectors created by the video encoder 312. In some aspects, the video encoder 312 can be built into a system-on-chip of the image capture device 300.

[0059] According to some aspects, the LECI frames generated by the embedded low power processor 304 can be stored in the storage module 308. In addition, the image and video data generated by the image processor 306 can also be stored in the storage module 308. The storage module 308 can be a built-in internal memory that can store a limited number of images and videos. According to some aspects, the storage module 308 can include a removable memory card, such as secure digital (SD), microSD, compact flash (CF), etc.

[0060] According to some aspects, the LECI frames generated by the embedded low-power processor 304 are stored in the storage module 308 and can be periodically transmitted to the media device 104 and / or uploaded to a cloud storage system of the multimedia environment 102. The user 132 can download the LECI to view the activities corresponding to the LECI frames in the monitored environment for the corresponding duration. According to some aspects, the image capture device can transmit the image and video data to the media device 104 and / or the cloud storage system for the generation of the LECI frames.

[0061] According to some aspects, when the image capture device 300 is power-limited (e.g., low power state, sleep state, low battery), the image capture device 300 may not be able to upload the LECI frames to the cloud. Instead, the image capture device 300 can provide an interface for directly accessing the LECI from the storage module 308 through a local network (e.g., a user WiFi network). According to some aspects, when the image capture device 300 is not power-limited, the image capture device 300 can capture video of the monitored environment for a duration and upload the video data to the media device 104 and / or the cloud storage system of the multimedia environment 102. According to some aspects, the LECI generation module 136 can generate one or more LECIs corresponding to the video data. The cloud storage system of the multimedia environment 102 can transmit the LECI frames to one or more media systems 102, which can provide an interface for viewing the LECI and determining the corresponding activity in the monitored environment for the corresponding duration.

[0062] As described above, the optical sensor 302 captures a sequence of image frames at a predetermined frame rate for a duration, and the embedded low power processor 304 or the image processor 306 combines the sequence of image frames to generate a LECI frame. According to some aspects, the frame rate at which the optical sensor 302 captures image frames can be predetermined based on a measure of activity in the monitored environment. When monitoring a dynamic environment with a large amount of activity, the optical sensor 302 of the image capture device 300 can be configured to record image frames at a higher frame rate to ensure that the captured sequence of image frames captures a sufficient amount of motion. Alternatively, the optical sensor 302 of the image capture device 300 can be configured to record image frames at a lower frame rate when monitoring an environment with a small amount of activity to ensure that the captured sequence of image frames captures a sufficient amount of motion. The image capture device 300 can be configured with software to perform motion detection on the captured sequence of image frames.

[0063] According to some aspects, the image capture device 300 can quantify activity metrics in the monitored environment based on comparing two or more captured images. For example, the image processor 306 of the image capture device 300 can be configured to quantify activity in the monitored environment using techniques such as optical flow, motion histograms, mean displacement vectors, mean square error techniques, etc.

[0064] According to some aspects, when the image capture device 300 determines that the activity metric corresponding to the monitored environment exceeds a predetermined activity threshold, the optical sensor 302 can be configured to capture image frames at a first predetermined rate. Similarly, when the image capture device 300 determines that the activity metric is below the predetermined activity threshold, the optical sensor 302 can be configured to capture image frames at a second predetermined rate that is lower than the first predetermined rate. That is, the first determined rate (e.g., more than 30 image frames per minute) can be higher than the second predetermined rate (e.g., less than or equal to 30 image frames per minute).

[0065] As described above, each LECI frame provides a single-image visual summary of activity in the monitored environment over a duration. In some aspects, the image capture device 300 records images of the monitored environment at a fixed location so that the single-image visual summary of activity represents a summary of activity in the monitored environment from a single field of view. According to some aspects, the duration corresponding to each LECI frame (i.e., the time period of activity summarized by each LECI frame) can be varied based on a metric of activity in the monitored environment and / or based on a selectable user option. In some aspects, the duration corresponding to each LECI frame can determine the rate at which the image capture device 300 generates LECI frames. For example, the rate at which LECI frames are generated can be varied based on a metric of activity in the monitored environment.

[0066] According to some aspects, the image capture device 300 can determine that it is monitoring a dynamic environment with a large amount of activity. The image capture device 300 can be configured to generate LECI frames corresponding to shorter durations. This functionality can ensure that each LECI frame provides an uncluttered summary of the activity within the corresponding duration. In addition, when monitoring an environment with a small amount of activity, generating LECI frames corresponding to longer durations can be sufficient to ensure a uncluttered summary of the activity within the corresponding duration for each LECI frame.

[0067] According to some aspects, when the image capture device 300 determines that the activity metric corresponding to the monitored environment exceeds a predetermined activity threshold, the embedded low power processor 304 can be configured to generate a LECI frame corresponding to a shorter duration than when the activity metric is below the predetermined activity threshold (e.g., each LECI frame summarizes less than 4 minutes of activity). Similarly, when the image capture device 300 determines that the activity metric corresponding to the monitored environment is below the predetermined activity threshold, the embedded low power processor 304 can be configured to generate a LECI frame corresponding to a longer duration (e.g., each LECI frame summarizes more than 4 minutes of activity).

[0068] Figure 4A An exemplary long exposure composite image (LECI) is shown in accordance with some aspects of the present disclosure. Figure 4A The example LECI frame 400A provides a single-image visual summary of a dog running against a primarily static background (eg, mountains 406 and trees 408) over a period of time.

[0069] As described above, the embedded low power processor 304 and / or the image processor 306 can generate an example LECI frame by combining a sequence of image frames captured by the optical sensor 302. According to some aspects, in order to generate the LECI frame, the embedded low power processor 304 first aligns the sequence of image frames captured by the optical sensor 302, and then combines these image frame sequences to generate the LECI frame. According to some aspects, the sequence of image frames can be combined using a pixel-by-pixel weighted average and / or stacking method. In addition, the sequence of image frames includes a group of image frames captured continuously over a period of time. The weighted average can be performed on a subset of image frames or all image frames of the sequence of image frames.

[0070] According to some aspects, calculating a pixel-by-pixel weighted average for a set of image frames can involve calculating a weighted average pixel value for each pixel position across all image frames in the set of image frames. According to some aspects, the optical sensor 302 can capture image frames in an RGB format including red, green, and blue channels. Calculating a weighted average for a set of RGB image frames can involve calculating a pixel-by-pixel weighted average for each color channel.

[0071] According to some aspects, optical sensor 302 can capture image frames in a format (e.g., YUV, YCbCr, etc.) indicating luma values ​​and chroma values ​​for each pixel. Calculating a weighted average of such image frames can involve calculating a pixel-by-pixel weighted average of luma values ​​over a set of image frames while ignoring chroma values. Alternatively or additionally, calculating a weighted average of image frames can involve calculating a pixel-by-pixel weighted average of chroma values ​​over the set of image frames.

[0072] According to some aspects, when calculating the weighted average of the sequence of image frames, the embedded low power processor 304 can select the weights assigned to the pixels to highlight one or more moving objects in the monitored environment. Alternatively or additionally, the image processor 306 can generate the LECI frame by calculating the weighted average of the sequence of image frames. According to some aspects, the embedded low power processor 304 and / or the image processor 306 can determine that they lack sufficient processing power and / or power (i.e., due to low power) to generate the LECI frame, but can transmit the sequence of image frames to the media device 106. In this case, the media device 106 can receive the sequence of image frames and generate the LECI frame by calculating the weighted average of the sequence of image frames. In some aspects, the steps for generating the LECI frame can be performed by the embedded low power processor 304; in some aspects, these steps can be distributed between different devices (e.g., one or more of the embedded low power processor, the media device 106, one or more content servers 120, and one or more system servers 126).

[0073] According to some aspects, LECI frames can be generated with annotations to represent actions performed by one or more moving objects. For example, when performing a pixel-by-pixel weighted average on a sequence of image frames, newer pixels can be assigned higher weights than older pixels to obtain a streak effect in the LECI frame along the direction of motion. According to some aspects, when combining a sequence of image frames to generate a LECI frame, pixel colors can be modified based on pixel age so that newer image frames in the sequence have different colors and / or hues than older image frames. According to some aspects, when combining a sequence of image frames to generate a LECI frame, pixel colors can be modified to create a heat map of activity in the LECI frame. According to some aspects, creating a LECI frame that displays a heat map of activity can involve maintaining a pixel change histogram to indicate areas where the most motion occurs and having different colors (e.g., using a red hue for high activity areas).

[0074] According to some aspects, Figure 4AThe example LECI frame 400A can be generated by the embedded low power processor 304 and / or the image processor 306 by assigning higher weights to newer image frames and lower weights to older image frames when calculating the weighted average of the image frame sequence. Figure 4A The example LECI frame in shows a dog running from left to right on a static background, annotated in the form of a streak effect. Figure 4A The LECI frame of 404 shows an object 404 that is annotated differently (i.e., having a darker hue) than objects 402A-E. In some aspects, the annotation can indicate an object in the LECI frame that corresponds to an object in the most recent image frame in the sequence of image frames averaged to generate the LECI frame. In addition, Figure 4A The LECI frames of FIG. 4 show objects 402A-E with different annotations (i.e., with lighter tones) to indicate that objects 402A-E correspond to objects of older image frames in the sequence of image frames. In some aspects, objects 402A, 402B, 402C, 402D, and 402E can be annotated differently (e.g., with lighter tones). Figure 4A 402A can correspond to the oldest image frame, object 402B can correspond to the next oldest image frame, and so on.

[0075] In some aspects, other annotations to the LECI image frame 400A are also possible. For example, a blur effect, an added indicator (e.g., an arrow), and / or a heat map indication can be added to the LECI image frame 400A. The annotation can also be based on temporal information from an image frame within the image frame sequence. Once the embedded low power processor 304 and / or the image processor 306 detects and identifies a moving object from an image frame within the image frame sequence, for example, based on pixel changes from an older image frame to a newer image frame, the embedded low power processor 304 and / or the image processor 306 can blur the older object, add an arrow pointing in the direction from the older object to the newer object and oriented in that direction, and / or add a heat map to indicate a time period during which the detected object did not move.

[0076] Figure 4B Another exemplary long exposure composite image (LECI) frame is shown in accordance with aspects of the present disclosure. Figure 4B An example LECI frame 400B is shown, which is generated by the embedded low power processor 304 and / or the image processor 306 by performing moving object detection on a sequence of image frames and by assigning one or more moving objects (e.g., Figure 4B The pixels corresponding to the running dog in the figure are assigned higher weights and the pixel-by-pixel weighted average of the group of images is calculated to generate the image. Figure 4AAn example LECI frame is similar to, Figure 4B The example LECI frame 400B also provides a single image visual summary of a dog running against a primarily static background (eg, mountains 416 and trees 418).

[0077] According to some aspects, moving object detection involves identifying and tracking objects moving on a sequence of image frames. Moving object detection can be performed using techniques such as background subtraction, frame differencing, etc. According to some aspects, the embedded low-power processor 304 implementing the background subtraction method first uses one or more static image frames (i.e., image frames without moving objects) to create a static background model, and then detects moving objects by subtracting the background from one or more subsequent image frames. According to some aspects, the embedded low-power processor 304 implementing the frame difference method calculates the absolute or relative difference between the pixel values ​​of corresponding pixels in two or more consecutive image frames to identify areas where pixel values ​​have changed significantly, and the areas where the pixel values ​​of the image frames have changed indicate moving objects. According to some aspects, the embedded low-power processor 304 can calculate motion vectors to indicate the movement of pixels between consecutive image frames in the sequence of image frames. According to some aspects, the embedded low-power processor 304 can calculate motion vectors using an optical flow method that estimates the apparent displacement of pixels to create a vector indicating the direction of a moving object.

[0078] According to some aspects, once a moving object in the sequence of image frames is identified, the embedded low power processor 304 generates a LECI frame by calculating a weighted average of the sequence of image frames. According to some aspects, when calculating the weighted average of the sequence of image frames, the weights assigned to pixels can be selected to highlight one or more moving objects in the sequence of image frames. For example, when performing a pixel-by-pixel weighted average on the sequence of image frames, pixels corresponding to moving objects can be assigned a higher weight than pixels corresponding to a static background.

[0079] According to some aspects, Figure 4B The example LECI frame 400B can be generated by the embedded low power processor 304 and / or the image processor 306 by assigning higher weights to pixels corresponding to moving objects (i.e., a running dog) and lower weights to pixels corresponding to a static background and calculating a weighted average of the image frame sequence. Figure 4B In the LECI frame of FIG. 4 , the image of the object (e.g., object 412A-F) has a different annotation (i.e., has a darker tone) than the static background (i.e., the hills and trees in the background). Other types of annotations can be used to distinguish moving objects from static objects, including highlighting, text annotations, and coloring.

[0080] Figure 5is a flow chart of a method 500 for generating a long exposure synthetic image (LECI) frame from stored video data according to some embodiments. The method 500 can be performed by processing logic, which can include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps are required to perform the disclosure provided herein. In addition, some of the steps can be performed simultaneously, or in different steps. Figure 5 The sequences shown are performed in a sequence which will be understood by those of ordinary skill in the art.

[0081] First, refer to Figure 1 The method 500 is described with reference to the illustrated embodiment of the LECI generation module 136 , although the method 500 is not limited to this embodiment.

[0082] At 502, the LECI generation module 136 receives a request (e.g., from the user 132) to generate LECI frames from a video file (e.g., a previously recorded video). As described above, each LECI frame provides a single-image visual summary of activities that occurred over a duration. Thus, viewing the LECI frames corresponding to a video segment of a video recording provides a quick and efficient method to identify activities that may have occurred over the duration of the video segment.

[0083] In 504, the LECI generation module 136 samples the segments of the video file at a predetermined rate to generate a sequence of image frames. According to some aspects, the image capture device 300 can generate video data (i.e., a video file). When the image capture device 300 is not power-constrained, the image capture device 300 can capture a video of the monitored environment (a high frame rate video) for a duration and upload the video data (i.e., a video file) to the media device 104, the content server 120, and / or a cloud storage system of the multimedia environment 102. According to some aspects, the video file can be previously recorded video footage stored on the content server 120. According to some aspects, the video file can be previously recorded video footage stored on a device of the media system 104 and / or the storage module 308 of the image capture device 300. Alternatively or additionally, the video file can be previously recorded video footage stored on a cloud storage system of the multimedia environment 102.

[0084] According to some aspects, LECI generation module 136 can be configured to sample the video file at a rate based on the amount of activity in the video. According to some aspects, LECI generation module 136 can be configured to quantify the activity in the video using techniques such as motion detection, feature extraction, optical flow analysis, machine learning, etc. LECI generation module 136 can be configured to sample the video at a rate proportional to the amount of activity in the video.

[0085] According to some aspects, the LECI generation module 136 determines that the activity metric corresponding to the video exceeds a predetermined activity threshold, and the LECI generation module 136 samples the video at a first predetermined rate to generate image frames. Similarly, when the LECI generation module 136 determines that the activity metric is below the predetermined activity threshold, the LECI generation module 136 can be configured to sample the video at a second predetermined rate that is lower than the first predetermined rate to generate image frames. That is, the first determined rate can be higher than the second predetermined rate. According to some aspects, the duration of the video segment corresponding to each LECI frame (i.e., the activity time period summarized by each LECI frame) can be changed based on the activity metric in the video and / or based on a selectable user option. In some aspects, the LECI generation module 136 can be configured to change the activity time period summarized by each LECI frame based on the activity metric in the video.

[0086] At 504, the LECI generation module generates a LECI frame by combining the sequence of image frames. According to some aspects, the LECI generation module 136 can generate the LECI frame by calculating a weighted average of the sequence of image frames.

[0087] According to some aspects, when calculating the weighted average of the sequence of image frames, the LECI generation module 136 can select the weights assigned to the pixels, and can be selected to highlight one or more moving objects in the monitoring environment. For example, when performing a pixel-by-pixel weighted average on the sequence of image frames, the LECI generation module 136 assigns higher weights to newer pixels (i.e., pixels corresponding to the image frames sampled later) than to older pixels (i.e., pixels corresponding to the image frames sampled earlier) to obtain a stripe effect along the direction of motion in the LECI frame. Alternatively or additionally, when combining the sequence of image frames to generate the LECI frame, the LECI generation module 136 can modify the pixel color based on the pixel lifespan so that the later image frames in the sequence have a different color and / or hue than the earlier image frames. According to some aspects, when combining the sequence of image frames to generate the LECI frame, the LECI generation module 136 can modify the pixel color to create a heat map of activity in the LECI frame. According to some aspects, creating a LECI frame that displays a heat map of activity can involve maintaining a histogram of pixel changes to indicate areas where the most motion is occurring and coloring them differently (eg, using a red tint for high activity areas).

[0088] According to some aspects, the LECI generation module 136 can generate the image frame sequence by performing moving object detection and by assigning one or more moving objects (e.g., Figure 4BThe LECI generation module 136 may be configured to generate a LECI frame by assigning a higher weight to pixels corresponding to a running dog in the image frame sequence and calculating a pixel-by-pixel weighted average on the group of images. According to some aspects, the LECI generation module 136 may be configured to identify moving objects in the image frame sequence and generate a LECI frame by calculating a weighted average of the image frame sequence. When calculating the weighted average of the image frame sequence, the LECI generation module 136 may be configured to assign weights to pixels to highlight one or more moving objects in the image frame sequence. For example, when performing a pixel-by-pixel weighted average on the image frame sequence, the LECI generation module 136 may assign a higher weight to pixels corresponding to moving objects than to pixels corresponding to static backgrounds.

[0089] In 506, the LECI generation module 136 sends the LECI frame to the network device. For example, the LECI generation module 136 can send the LECI frame to the media device 106. The media device 106 can display an interface for viewing the received LECI frame. The user 132 can view the LECI frame using the interface of the media device 106 to identify activities that may occur in the video.

[0090] Figure 6 6 is a flow chart of a method 600 for generating a long exposure synthetic image (LECI) frame from a sequence of image frames according to some embodiments. The method 600 can be performed by processing logic, which can include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps are required to perform the disclosure provided herein. In addition, some of the steps can be performed simultaneously, or in different steps. Figure 6 The sequences shown are performed in a sequence which will be understood by those of ordinary skill in the art.

[0091] First, refer to Figure 3 The method 600 is described with reference to the illustrated embodiment of the image capture device 300 , although the method 600 is not limited to this embodiment.

[0092] In 602, the image capture device 300 captures a sequence of image frames over a duration. According to some aspects, the sequence of images is captured using an optical sensor 302 of the image capture device 300. According to some aspects, the optical sensor 302 generates uncompressed raw pixel data corresponding to the captured image and / or video. The optical sensor 302 can capture color information by using a pattern of red, green, and blue (RGB) filters on pixels. Alternatively or additionally, the optical sensor 302 can capture the brightness (Y) and chrominance (UV) components of a pixel. The uncompressed raw pixel data generated by the optical sensor 302 can be sent to the embedded low-power processor 304 and / or the image processor 306 for further processing (e.g., quantization and compression).

[0093] According to some aspects, the optical sensor 302 captures a sequence of image frames at a predetermined frame rate over a duration. According to some aspects, the frame rate at which the optical sensor 302 captures image frames can be predetermined based on a measure of activity in the monitored environment. When monitoring a dynamic environment with a large amount of activity, the optical sensor can be configured to record image frames at a higher frame rate to ensure that the captured sequence of image frames captures a sufficient amount of motion. Alternatively, the optical sensor 302 can be configured to record image frames at a lower frame rate when monitoring an environment with a small amount of activity. In addition, the image capture device 300 can be configured with software to perform motion detection on the captured sequence of image frames.

[0094] According to some aspects, the image capture device 300 can quantify activity metrics in the monitored environment based on comparing two or more captured images. For example, the image capture device 300 can use techniques such as optical flow, motion histograms, mean displacement vectors, mean square error techniques, etc. to quantify activity in the monitored environment.

[0095] According to some aspects, the image capture device 300 captures a sequence of image frames at a capture rate that is proportional to the activity metric in the monitored environment. In addition, the image capture device 300 can capture a sequence of image frames for a duration that is inversely proportional to the activity metric in the monitored environment. When the image capture device 300 determines that the activity metric corresponding to the monitored environment exceeds a predetermined activity threshold, the optical sensor 302 can be configured to capture image frames at a first predetermined rate. Similarly, when the image capture device 300 determines that the activity metric is below a predetermined activity threshold, the optical sensor 302 can be configured to capture image frames at a second predetermined rate that is lower than the first predetermined rate. That is, the first determined rate can be higher than the second predetermined rate.

[0096] In 604, the image capture device 300 generates a LECI frame by combining the sequence of image frames. According to some aspects, the embedded low power processor 304 coupled to the optical sensor 302 generates the LECI frame by combining the sequence of image frames. Alternatively or additionally, the image processor 306 of the image capture device 300 can generate the LECI frame by combining the sequence of image frames. According to some aspects, the embedded low power processor 304 and / or the image processor 306 can generate the LECI frame by calculating a weighted average of the sequence of image frames.

[0097] According to some aspects, when calculating the weighted average of the sequence of image frames, the embedded low power processor 304 can select the weights assigned to the pixels, and can be selected to highlight one or more moving objects in the monitoring environment. For example, when performing a pixel-by-pixel weighted average on the sequence of image frames, the embedded low power processor 304 assigns a higher weight to the newer pixels than the older pixels to obtain a stripe effect along the direction of motion in the LECI frame. Alternatively or additionally, when combining the sequence of image frames to generate the LECI frame, the embedded low power processor 304 can modify the pixel color according to the pixel life so that the newer image frames in the sequence have different colors and / or hues than the older image frames. According to some aspects, when combining the sequence of image frames to generate the LECI frame, the embedded low power processor 304 can modify the pixel color to create a heat map of activity in the LECI frame. According to some aspects, creating a LECI frame that displays a heat map of activity can involve maintaining a pixel change histogram to indicate the area where the most motion occurs, and making it have a different color (e.g., using a red hue for high activity areas).

[0098] According to some aspects, the embedded low-power processor 304 and / or the image processor 306 can detect moving objects by performing moving object detection on a sequence of image frames and by assigning one or more moving objects (e.g., Figure 4B The embedded low-power processor 304 can assign higher weights to pixels corresponding to a running dog in the image frame sequence, and calculate a pixel-by-pixel weighted average on the group of images to generate a LECI frame. According to some aspects, the embedded low-power processor 304 can identify moving objects in the image frame sequence and generate a LECI frame by calculating a weighted average of the image frame sequence. When calculating the weighted average of the image frame sequence, the embedded low-power processor 304 can assign weights to pixels to highlight one or more moving objects in the image frame sequence. For example, when performing pixel-by-pixel weighted averaging on the image frame sequence, the embedded low-power processor 304 assigns higher weights to pixels corresponding to moving objects than to pixels corresponding to a static background.

[0099] In 606, the image capture device 300 transmits the LECI frame to the network device using a transceiver coupled to the embedded low power processor. According to some aspects, the network device can be a media device 106, a content server 120, and / or a cloud-based storage device of the media system 104. According to some aspects, the image capture device 300 can store the LECI frame generated by the embedded low power processor 304 in the storage module 308 and can be periodically transmitted to the media device 104 and / or uploaded to the cloud storage system of the multimedia environment 102.

[0100] According to some aspects, when the image capture device 300 is power-limited (e.g., a low power state, a sleep state, or low battery), the image capture device 300 may not be able to upload LECI frames to the cloud. Instead, the image capture device 300 can provide an interface for directly accessing the LECI from the storage module 308 through a local network (e.g., a user WiFi network). Alternatively, when the image capture device 300 is not power-limited, the image capture device 300 can capture video of the monitored environment for a duration and upload the video data to the cloud storage system of the media device 104 and / or the multimedia environment 102. The cloud-based processing module can generate one or more LECIs corresponding to multiple durations from the uploaded video data.

[0101] Figure 7 is a flow chart of a method 700 for identifying movement of large objects between frames of video data, according to some embodiments. The method 700 can be performed by processing logic, which can include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps are required to perform the disclosure provided herein. In addition, some of the steps can be performed simultaneously, or in different steps. Figure 7 The sequences shown are performed in a sequence which will be understood by those of ordinary skill in the art.

[0102] First, refer to Figure 3 The method 700 is described with reference to the illustrated embodiment of the video encoder 312 and the central processor 314, although the method 700 is not limited to this embodiment.

[0103] At 702, the video encoder 312 of the image capture device 300 receives video data from the optical sensor 302. The video encoder 312 can be embedded on a system on a chip (SoC) of the image capture device 300. The video encoder 312 can be configured to receive raw video data from the optical sensor 302 and convert it to a digital format that can be streamed, recorded, or transmitted over a network. According to some aspects, the video encoder 312 can be configured to compress the video data using compression standards such as H.264, H.265 (HEVC), and VP9.

[0104] At 704, video encoder 312 processes a sequence of video frames (e.g., image frames) of video data to generate a compressed bitstream including clusters of motion vectors corresponding to one or more identified motion patterns in the video data. In some aspects, video encoder 312 can consume less power to perform certain video processing operations (e.g., generating motion vectors) than central processor 314. In some aspects, to improve power efficiency, video encoder 312 can be configured to process a sequence of image frames of video data to generate motion vectors.

[0105] According to some aspects, motion vectors can be generated by comparing consecutive video frames in a sequence of video frames. According to some aspects, video encoder 312 can calculate motion vectors using an optical flow method that estimates the apparent displacement of pixels to create a vector indicating the direction of a moving object. According to some aspects, motion vectors can be generated using a block matching algorithm. According to some aspects, motion vectors in the compressed video data generated by video encoder 312 can be scanned by low power central processor 312 to identify the movement of large objects between frames of video data.

[0106] At 706, the motion vectors generated by the video encoder 312 are processed to detect moving objects in the surveillance environment corresponding to the video data. Therefore, large objects in motion in the video data can be identified in an energy-saving manner by comparing consecutive video frames using the on-chip video encoder 312 and using the motion vectors scanned by the central processor 312. According to some aspects, the low-power central processor 314 can identify large objects in motion by performing motion vector clustering to group motion vectors that can correspond to the movement of large objects within the video sequence. In some aspects, the motion vectors can be visualized on the video data (e.g., as arrows indicating the direction of movement). The central processor 314 can cluster the motion vectors using a clustering algorithm (e.g., k-means clustering, hierarchical clustering, and density-based spatial clustering).

[0107] Example Computer System

[0108] For example, using one or more well-known computer systems (e.g., Figure 7 The computer system 700 shown in the figure can be used to implement various embodiments. For example, the media device 106 can be implemented using a combination or sub-combination of computer systems 700. Similarly or alternatively, for example, one or more computer systems 700 can be used to implement any of the embodiments discussed herein, as well as combinations and sub-combinations thereof. The computer system 700 can include one or more processors (also referred to as central processing units or CPUs), such as processor 704. The processor 704 can be connected to a communication infrastructure or bus 706.

[0109] The computer system 700 can also include one or more user input / output devices 703 (eg, a display, keyboard, pointing device, etc.) that can communicate with a communication infrastructure 706 via one or more user input / output interfaces 702 .

[0110] One or more processors 704 can be a graphics processing unit (GPU). In an embodiment, a GPU can be a processor that is a dedicated electronic circuit designed to process mathematically intensive applications. A GPU can have a parallel structure that efficiently processes large blocks of data in parallel, such as mathematically intensive data common to computer graphics applications, images, videos, etc.

[0111] The computer system 700 can also include a main memory or primary storage 708, such as a random access memory (RAM). The main memory 708 can include one or more levels of cache. The main memory 708 can store control logic (ie, computer software) and / or data therein.

[0112] The computer system 700 can also include one or more secondary storage devices or memories 710. The secondary storage 710 can include, for example, a hard disk drive 712 and / or a removable storage device or drive 714. The removable storage drive 714 can be a floppy disk drive, a tape drive, an optical drive, an optical storage device, a tape backup device, and / or any other storage device / drive.

[0113] The removable storage drive 714 can interact with a removable storage unit 718. The removable storage unit 718 can include a computer usable or readable storage device having computer software (control logic) and / or data stored thereon. The removable storage unit 718 can be a floppy disk, a magnetic tape, a compact disk, a DVD, an optical storage disk, and / or any other computer data storage device. The removable storage drive 714 can read from and / or write to the removable storage unit 718.

[0114] The secondary memory 710 can include other means, devices, components, tools, or other methods to allow computer programs and / or other instructions and / or data to be accessed by the computer system 700. These means, devices, components, tools, or other methods can include, for example, a removable storage unit 722 and an interface 720. Examples of a removable storage unit 722 and an interface 720 can include a program cartridge and cartridge interface (e.g., as found in a video game device), a removable storage chip (e.g., an EPROM or PROM) and associated socket, a memory stick and USB or other port, a memory card and associated memory card slot, and / or any other removable storage unit and associated interface.

[0115] The computer system 700 can also include a communication or network interface 724. The communication interface 724 enables the computer system 700 to communicate and interact with any combination of external devices, external networks, external entities, etc. (individually and collectively referenced by reference numeral 728). For example, the communication interface 724 can allow the computer system 700 to communicate with an external or remote device 728 via a communication path 726, which can be wired and / or wireless (or a combination thereof) and can include any combination of a local area network (LAN), a wide area network (WAN), the Internet, etc. Control logic and / or data can be transmitted to and from the computer system 700 via the communication path 726.

[0116] To name a few non-limiting examples, the computer system 700 can also be a personal digital assistant (PDA), a desktop workstation, a notebook or laptop computer, a netbook, a tablet computer, a smart phone, a smart watch or other wearable device, an appliance, part of the Internet of Things and / or an embedded system, etc., or any combination thereof.

[0117] Computer system 700 can be a client or server, accessing or hosting any applications and / or data through any delivery paradigm, including but not limited to remote or distributed cloud computing solutions; local or on-site software ("on-site" cloud-based solutions); "as a service" models (e.g., content as a service (CaaS), digital content as a service (DCaaS), software as a service (SaaS), hosted software as a service (MSaaS), platform as a service (PaaS), desktop as a service (DaaS), framework as a service (FaaS), backend as a service (BaaS), mobile backend as a service (MBaaS), infrastructure as a service (IaaS), etc.); and / or a hybrid model including any combination of the above examples or other services or delivery paradigms.

[0118] Any applicable data structures, file formats, and schemas in computer system 700 can be derived from standards, including but not limited to JavaScript Object Notation (JSON), Extensible Markup Language (XML), Yet Another Markup Language (YAML), Extensible Hypertext Markup Language (XHTML), Wireless Markup Language (WML), MessagePack, XML User Interface Language (XUL), or any other functionally similar representation, used alone or in combination. Alternatively, proprietary data structures, formats, or schemas can be used alone or in combination with known or open standards.

[0119] In some embodiments, a tangible non-transitory device or article of manufacture including a tangible non-transitory computer usable or readable medium having control logic (software) stored thereon can also be referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system 700, main memory 708, auxiliary memory 710, and removable storage units 718 and 722, and tangible articles of manufacture embodying any combination of the foregoing. Such control logic, when executed by one or more data processing devices (e.g., computer system 700 or one or more processors 704), can cause such data processing devices to operate as described herein.

[0120] Based on the teachings contained in this disclosure, how to use Figure 7 It will be apparent to those skilled in the relevant art that data processing devices, computer systems, and / or computer architectures other than those shown can be used to make and use the embodiments of the present disclosure. In particular, the embodiments can be implemented to run with software, hardware, and / or operating systems other than those described herein.

[0121] in conclusion

[0122] It should be understood that the detailed description section, but not other sections, is intended to be used to interpret the claims. The other sections may set forth one or more but not all exemplary embodiments contemplated by the inventors, and therefore are not intended to limit the present disclosure or the appended claims in any way.

[0123] Although the present disclosure describes exemplary embodiments for exemplary fields and applications, it should be understood that the present disclosure is not limited thereto. Other embodiments and modifications thereof are feasible and within the scope and spirit of the present disclosure. For example, without limiting the generality of this paragraph, the embodiments are not limited to the software, hardware, firmware and / or entities illustrated and / or described herein. In addition, the embodiments (whether or not explicitly described herein) have significant utility for fields and applications beyond the examples described herein.

[0124] Embodiments are described herein with the aid of functional building blocks that illustrate the implementation of specified functions and their relationships. For ease of description, the boundaries of these functional building blocks are arbitrarily defined herein. As long as the specified functions and relationships (or their equivalents) are properly performed, other boundaries can be defined. In addition, alternative embodiments can use sequential execution function blocks, steps, operations, methods, etc. different from those described herein.

[0125] "One embodiment", "embodiment", "exemplary embodiment" or similar phrases mentioned herein indicate that the described embodiment can include specific features, structures or characteristics, but not every embodiment necessarily includes specific features, structures or characteristics. In addition, these phrases do not necessarily refer to the same embodiment. In addition, when describing specific features, structures or characteristics in conjunction with an embodiment, within the knowledge of a technician in the relevant field, the technician in the relevant field can combine the features, structures or characteristics into other embodiments, regardless of whether they are explicitly mentioned or described in this article. In addition, some embodiments can be described using the expressions "coupled" and "connected" and their derivatives. These terms are not necessarily synonymous with each other. For example, some embodiments can use the terms "connected" and / or "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" can also mean that two or more elements are not in direct contact with each other, but still collaborate or interact with each other.

[0126] The breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

1. A computer-implemented method for generating a long exposure composite image (LECI) frame, wherein: The LECI frame provides a summary of activity in a monitored environment over a duration, the computer-implemented method comprising: capturing, by at least one computer processor of an image capture device, a sequence of image frames over the duration using an optical sensor of the image capture device; generating the LECI frame by combining the sequence of image frames using an embedded low power processor coupled to the optical sensor and dedicated to generating the LECI frame; and The LECI frame is transmitted to a network device using a transceiver coupled to the embedded low power processor.

2. The computer-implemented method of claim 1 , wherein: The sequence of image frames includes a first set of image frames and a second set of image frames, wherein the first set of image frames is associated with a first time period within the duration and the second set of image frames is associated with a second time period within the duration, wherein the first time period is more recent than the second time period, and wherein generating the composite image frame includes: The weighted averaging of the image frame sequence is performed in the following manner: assigning a first set of weights to a first set of image frames of the sequence of image frames; and A second set of weights is assigned to the second set of image frames, wherein each of the first set of weights is greater than all of the weights of the second set of weights.

3. The computer-implemented method of claim 1 , wherein: Generating the LECI frame includes: The sequence of image frames is combined by calculating a pixel-by-pixel weighted average of pixel values ​​over the sequence of image frames.

4. The computer-implemented method of claim 1 , wherein: Generating the LECI frame includes: identifying a moving object in one or more image frames of the sequence of image frames; and Weighted averaging is performed by assigning higher weights to pixels corresponding to moving objects in one or more image frames.

5. The computer-implemented method of claim 1 , wherein: Generating the LECI frame includes: A relative displacement of one or more pixels between a set of image frames of the sequence of image frames is identified to generate one or more direction vectors indicating a direction of motion of an object in the sequence of image frames.

6. The computer-implemented method of claim 1 , wherein: Capturing the image frame sequence comprises: The sequence of image frames is captured at a capture rate proportional to a measure of activity in the monitored environment.

7. The computer-implemented method of claim 1 , wherein: The duration is inversely proportional to a measure of activity in the monitored environment.

8. The computer-implemented method of claim 1 , wherein: The image capture device is a battery powered camera.

9. A system for generating a long exposure composite image (LECI) frame, wherein: The LECI frame provides a summary of activities in a monitored environment over a duration, the system comprising: one or more memories; and at least one processor, each coupled to at least one of the memories of the image capture device and configured to perform operations comprising: capturing a sequence of image frames over the duration using an optical sensor of the image capture device; generating the LECI frame by combining the sequence of image frames using an embedded low power processor coupled to the optical sensor and dedicated to generating the LECI frame; and The LECI frame is transmitted to a network device using a transceiver coupled to the embedded low power processor.

10. The system according to claim 9, wherein: The sequence of image frames includes a first set of image frames and a second set of image frames, wherein the first set of image frames is associated with a first time period within the duration and the second set of image frames is associated with a second time period within the duration, wherein the first time period is more recent than the second time period, and wherein generating the composite image frame includes: The weighted averaging of the image frame sequence is performed in the following manner: assigning a first set of weights to a first set of image frames of the sequence of image frames; and A second set of weights is assigned to the second set of image frames, wherein each of the first set of weights is greater than all of the weights of the second set of weights.

11. The system according to claim 9, wherein: Generating the LECI frame includes: The sequence of image frames is combined by calculating a pixel-by-pixel weighted average of pixel values ​​over the sequence of image frames.

12. The system according to claim 9, wherein: Generating the LECI frame includes: identifying a moving object in one or more image frames of the sequence of image frames; and Weighted averaging is performed by assigning higher weights to pixels corresponding to moving objects in one or more image frames.

13. The system according to claim 9, wherein: Generating the LECI frame includes: A relative displacement of one or more pixels between a set of image frames of the sequence of image frames is identified to generate one or more direction vectors indicating a direction of motion of an object in the sequence of image frames.

14. The system according to claim 9, wherein: Capturing the image frame sequence comprises: The sequence of image frames is captured at a capture rate proportional to a measure of activity in the monitored environment.

15. The system according to claim 9, wherein: The duration is inversely proportional to a measure of activity in the monitored environment.

16. A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations for generating a long exposure synthetic image (LECI) frame, wherein: The LECI frame provides a summary of activities in the monitored environment over a duration, and the operations include: capturing, by at least one computer processor of an image capture device, a sequence of image frames over the duration using an optical sensor of the image capture device; generating the LECI frame by combining the sequence of image frames using an embedded low power processor coupled to the optical sensor and dedicated to generating the LECI frame; and The LECI frame is transmitted to a network device using a transceiver coupled to the embedded low power processor.

17. The non-transitory computer readable medium of claim 16, wherein: The sequence of image frames includes a first set of image frames and a second set of image frames, wherein the first set of image frames is associated with a first time period within the duration and the second set of image frames is associated with a second time period within the duration, wherein the first time period is more recent than the second time period, and wherein generating the composite image frame includes: The weighted averaging of the image frame sequence is performed in the following manner: assigning a first set of weights to a first set of image frames of the sequence of image frames; and A second set of weights is assigned to the second set of image frames, wherein each of the first set of weights is greater than all of the second set of weights.

18. The non-transitory computer readable medium of claim 16, wherein: Generating the LECI frame includes: The sequence of image frames is combined by calculating a pixel-by-pixel weighted average of pixel values ​​over the sequence of image frames.

19. The non-transitory computer readable medium of claim 16, wherein: Generating the LECI frame includes: identifying a moving object in one or more image frames of the sequence of image frames; and Weighted averaging is performed by assigning higher weights to pixels corresponding to moving objects in one or more image frames.

20. The non-transitory computer readable medium of claim 16, wherein: Capturing the image frame sequence comprises: The sequence of image frames is captured at a capture rate proportional to a measure of activity in the monitored environment.

Images