Alert generation based on event detection in a video feed

Abstract

This invention relates to alarm generation based on event detection in video feeds. Methods, systems, and computer program products for processing streams of image frames captured by a camera system are also included. In response to the detection of an event, a hard-coded alarm image frame is generated. The hard-coded alarm image frame includes motion increments and / or color changes relative to the event image frame. A stream of encoded image frames is generated, wherein the hard-coded alarm image frame is inserted into the stream of encoded image frames after the encoded event image frame in display order.

Description

TECHNICAL FIELD

[0001] The present invention relates to a video surveillance system, and more specifically to generating an alarm to an operator in response to detecting an event in a video feed. BACKGROUND

[0002] In many surveillance systems, video or other data from a large number of cameras and / or other sensors is managed and displayed on surveillance screens in an operator control room. Typically, the operator control room has several screens, such as 3 to 6 screens. Each screen displays several video feeds, e.g. 4x4 video feeds, which are monitored by the operator. Thus, the operator has to pay attention to 48 (3x16) to 96 (6x16) video feeds at the same time in order to be able to detect an event in a single video feed.

[0003] An event can be a movement of an object such as a vehicle, an intruder into a restricted area, a detected face, a crowded area, just to give a few examples. However, due to the large number of video feeds, there can be a risk that the operator misses detecting an event in one of the video feeds, especially if the event is subtle or happens very quickly, while the operator's attention can be temporarily diverted from the specific video feed containing the event. Thus, there is a need to direct the operator's attention to a specific video feed when an event occurs in the video feed.

[0004] A video surveillance system is described in WO 2006 / 006081. The video surveillance system consists of intelligent cameras, servers and clients connected through an IP network in a wired or wireless configuration. The system is designed to protect the privacy of the people and the goods under surveillance. The operation of the intelligent cameras is optimized to trade-off between perceptible visual quality of decoded video and power consumption. The servers receive, store, manage and distribute video sequences on wired and wireless channels to various clients and users with different device capabilities, channel characteristics and preferences. The use of seamless scalable coding of the video sequences prevents the need for transcoding operations at any point in the system.

[0005] US2003 / 0122667 describes a system and method for enhancing security at self-checkout stations. The system includes a security agent application running in terminals at several self-checkout stations. The security agent software generates event messages about security events occurring at the station and transmits these event messages to a server. The server sends priority event messages as alarm messages to a security controller. The security controller is coupled to security cameras, image data storage devices, and image data display devices, and generates control messages for these devices based on received alarm messages. The control messages for the security cameras operate the cameras to zoom, focus, tilt, or pan in response to events occurring at the station. The controller may insert visible alarm indicators into the video stream of a camera pointing at a monitor or insert audible tones into the audio stream of the video stream to alert security personnel to the display of a security event occurring at the station. Summary of the Invention

[0006] According to a first aspect, the present invention relates to a method for processing a stream of image frames in a camera system. The method includes:

[0007] In response to the detection of an event, a hard-coded alarm image frame is generated, wherein the hard-coded alarm image frame is an inter-frame image frame and includes motion increments and / or color changes relative to the event image frame, and wherein the hard-coded alarm image frame is generated in software to produce the desired changes in the video stream when displayed to an operator; and

[0008] A stream of encoded image frames is generated, wherein hard-coded alarm image frames are inserted into the stream of encoded image frames after the encoded event image frames in display order.

[0009] This method makes it easier for operators to receive alerts about video feeds in monitoring situations where suspicious events may occur. The method is not limited by the type of event and / or detection mechanism and can be applied to any type of event. The event image frame used herein is an event-related image frame. An event image frame can be an image frame that includes the actual event, or an image frame captured at or before / after the event. Further, when used in this disclosure, the expression "hard-coded alarm image frame" should be understood as an alarm image frame generated in software to produce a desired change in the video when displayed to the operator, thereby alerting the operator to the event. Therefore, it should be understood that the hard-coded alarm image frame is not generated by an encoder through encoding image data. Since the alarm image frame is hard-coded, it does not need to be encoded by an encoder but can be directly inserted into the image stream output from the encoder. The desired change in the video when displayed to the operator can be jitter and / or color changes in the displayed video to alert the operator. The type of alarm (i.e., "jitter" and / or color change) is configurable, allowing the selection of the most appropriate alarm type based on the current situation.

[0010] As is known to those skilled in the art, video coding formats specify temporal video compression implemented in terms of intra-frame and inter-frame image frames. The encoded inter-frame image frame includes: 1) a reference to a reference frame, i.e., an image frame used for inter-frame prediction when the decoder decodes the encoded inter-frame image frame; 2) a frame number so that the decoder can decode the inter-frame image frames in the correct decoding order; and 3) an indication of display order so that the decoder can display the decoded inter-frame image frame at the correct time position in the decoded video stream. Therefore, those skilled in the art will understand that a hard-coded alarm image frame as an inter-frame image frame includes: 1) a reference to an event image frame used when decoding the hard-coded alarm image frame, the hard-coded alarm image frame including motion increments and / or color changes relative to the event image frame; 2) a frame number indicating the decoding order of the hard-coded alarm image frames to be decoded; and 3) an indication of display order so that the decoder can display the decoded hard-coded alarm image frames in the correct display order to produce the desired changes in the video stream when displayed to an operator. In this disclosure, sometimes hard-coded alarm image frames are simply referred to as alarm image frames.

[0011] According to one embodiment, the method further includes encoding the event image frame as a non-display frame. While it is possible to alert the operator without encoding the event image frame as a non-display frame, the visual effect is generally more pleasing if the event image frame is encoded as a non-display image frame. Furthermore, by encoding the event image frame as a "non-display" image frame and inserting the alert image frame after the corresponding event image frame (i.e., in the display order, even though the event image frame is a non-display image frame and therefore not displayed, the alert image frame should be inserted after the event image frame as if the event image frame were to be displayed), the original display frame rate can be preserved, thus giving the video on the operator's monitor a natural look. "Non-display" in this document means that the image frame is not shown to the user. The H.265 encoding standard (and other newer encoding standards, such as Google's VP10) allows frames to be marked as "non-display." For example, in H.265, this can be done by setting the pic_output_flag in the slice header to false or setting the no_display flag in the SEI header to true.

[0012] According to one embodiment, the event is either an event detected in an event image frame within a stream of image frames, or an external event. That is, as described above, the method is not limited to any particular type of event detection. For example, an event can be detected by analyzing the content of an image frame (e.g., by a camera system analyzing the content of an image frame), but it can also be triggered by an external sensor (such as a door being opened) and associated with an event image frame captured at (or close to) the time the sensor is triggered. This significantly increases the possibilities for using the method according to the invention.

[0013] According to one embodiment, the generation steps are repeated twice for a number of event image frames. Even though the method described herein is used for only a single event image frame, it still provides an improvement over existing techniques in alerting the operator to an event. However, by repeating the generation steps of the method for a number of event image frames, a “jittering” or “flickering” appearance can be achieved in the image stream, making it easier for the operator to be alerted to an event that has occurred in a particular video feed. The jittering / flickering can last for a certain amount of time, a predetermined amount of time, or until the operator confirms the alarm.

[0014] According to one embodiment, motion increments include motion relative to the event image frame in the horizontal, vertical, and any combination thereof directions. Most encoders are capable of applying both horizontal and vertical motion. Therefore, it should be understood that alarm image frames generated in software, for example by the image processing pipeline (i.e., hard-coded alarm image frames), can also include motion increments containing both horizontal and vertical motion. This allows images in the image stream to be moved (and thus “jittered”) in virtually any pattern desired by the user. In some cases, these patterns can also be associated with specific types of events. For example, the entry of a suspicious person into a scene might cause “small vertical jitter,” while someone attempting to break into a building might cause “bright red flashes” or similar.

[0015] According to one embodiment, the motion increment has a configurable size or a predefined size. The ability to select a predefined or configurable size of motion increases the versatility of the invention and makes it suitable for many different users. For example, the size of the motion increment can be related to various characteristics of the event, to name just a few, such as type, intensity, importance, or accuracy.

[0016] According to one embodiment, motion increments are applied only to a portion of the alarm image frame. That is, in some cases, it may not be necessary to apply motion to the entire alarm image frame. For example, if an object in the scene is leaning against a wall of a uniform color, "shaking" the entire wall is almost meaningless; shaking only the object is sufficient, thus saving computational resources while still allowing the operator to be alerted.

[0017] According to one embodiment, the color variation includes one or more of the following: a representation of more colors relative to the event image frame, a representation of fewer colors relative to the event image frame, and a color variation relative to the event image frame. This allows for any color configuration within the alarm image frame and can create various effects such as more or less intense flashing, alternating color flashing, or converting the entire alarm image frame to a uniform color.

[0018] According to one embodiment, the color change is applied only to a portion of the alarm image frame. Similar to the description above, in some cases, applying the color change only to a specific object of interest may be sufficient, highlighting that object while leaving the rest of the alarm image frame unchanged. This could be useful, for example, in face recognition scenarios where, within a group of people, the operator may only be interested in a specific individual. Furthermore, the color change does not necessarily need to be applied to an object, and specific portions of the alarm image frame (to give just one example, such as the boundaries surrounding the alarm image frame) can be colored, or the alarm image frame can be made to have stripes or some other pattern, which can be particularly useful if the specific alarm pattern is associated with a specific event type.

[0019] According to one embodiment, the method further includes, in response to operator input, removing the hidden state of the event image frame and changing the state of the alarm image frame to hidden, so that the operator can view the event captured by the camera system. That is, after the operator has received and confirmed the alarm, she can view the raw image captured by the camera to determine what the event is and whether any action is required.

[0020] According to one embodiment, the alarm image frame is one of a forward prediction frame (P-frame) containing motion increments relative to the event image frame and a bidirectional frame (B-frame) containing motion increments relative to the event image frame. In the context of image processing, P-frames and B-frames are well-known standard concepts, thus the invention described herein can be readily integrated with existing and future monitoring systems.

[0021] According to one embodiment, a hard-coded alarm image frame is generated based on an alarm image frame generated outside the camera system. Therefore, the generation of a hard-coded alarm image frame can include generating a hard-coded alarm image frame based on an alarm image frame generated outside the camera system. This reduces the computing power required by the camera and makes the invention applicable to a wider range of devices. For example, the alarm image frame can be generated by an external application that neither controls the camera image processing pipeline nor the video management system (VMS) event processing system. For example, the external application can be triggered by external devices such as microphones, radar, passive infrared (PIR) sensors, motion sensors, door sensors, window sensors, etc. The alarm image frame generated by the external application includes motion increments and / or color changes, such as predefined or predetermined motion increments and / or color changes. Further, the alarm image frame can be input into an image processing pipeline that can generate a hard-coded alarm image frame based on an externally generated alarm image frame. Since the hard-coded alarm image frames are generated relative to the event image frames, it should be understood that the image processing pipeline generates inter-frame image frames from the hard-coded alarm image frames. These inter-frame image frames include motion increments and / or color changes from the input alarm image frames, as well as a reference to the event image frames. As mentioned above and as is known to those skilled in the art, the inter-frame image frames also include indications of frame numbering and display order.

[0022] According to one embodiment, the event image frame is a reference frame for a group of pictures (GOP). In the context of image processing, GOP is a well-known standard concept, thus the invention described herein can be readily integrated with existing and future monitoring systems.

[0023] According to a second aspect, the present invention relates to a camera system. The camera system includes a lens, an image sensor, an image processing pipeline, and an encoder. The lens and image sensor are configured to capture a stream of image frames. The image processing pipeline is configured to generate hard-coded alarm image frames in response to the detection of an event, wherein the hard-coded alarm image frames are inter-frame image frames and include motion increments and / or color changes relative to the event image frames, and wherein the hard-coded alarm image frames are generated in software to produce desired changes in the video stream when displayed to an operator. The encoder is configured to generate a stream of encoded image frames, wherein the hard-coded alarm image frames are inserted into the stream of encoded image frames after the encoded event image frames in display order.

[0024] The advantages of a system correspond to the advantages of a method and can vary similarly.

[0025] According to a third aspect, the present invention relates to a computer program product for processing a stream of image frames captured by a camera system. The computer program includes instructions corresponding to the following steps:

[0026] In response to the detection of an event, a hard-coded alarm image frame is generated, wherein the hard-coded alarm image frame is an inter-frame image frame and includes motion increments and / or color changes relative to the event image frame, and wherein the hard-coded alarm image frame is generated in software to produce the desired changes in the video stream when displayed to an operator; and

[0027] A stream of encoded image frames is generated, wherein hard-coded alarm image frames are inserted into the stream of encoded image frames after encoded event image frames.

[0028] Computer programs involve advantages corresponding to the advantages of methods and can vary similarly.

[0029] Details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features and advantages of the invention will be apparent from the specification, drawings, and claims. Attached Figure Description

[0030] Figure 1 This is a schematic diagram illustrating an exemplary environment 100 in which various methods and systems of the present invention can be applied according to one embodiment.

[0031] Figure 2 This is a schematic diagram illustrating a display with four video feeds according to one embodiment.

[0032] Figure 3 This illustrates an embodiment. Figure 1 A block diagram showing a detailed view of the camera system 108 shown in the figure.

[0033] Figure 4 This is a flowchart illustrating a method for processing a stream of image frames according to one embodiment.

[0034] Figure 5 This is a schematic diagram of a display showing four video feeds according to one embodiment, in which one video feed is modified to attract the operator's attention.

[0035] The same reference numerals in the various figures denote the same elements. Detailed Implementation

[0036] As described above, one objective of various embodiments of the present invention is to provide an improved technique for directing an operator's attention to a video feed in the event of an event occurring within that feed. The exact nature of an "event" is not within the scope of this invention and can be determined individually. However, as mentioned in the summary section of the specification, examples of events include the movement of objects such as vehicles, intruders in restricted areas, detected faces, crowded areas, etc. These can all be considered "visible" events. However, invisible events may also exist, such as sounds or events of damage or malfunction, but these can also be handled according to the techniques described below. Various embodiments of the present invention relate to what happens after an event is detected.

[0037] According to the various embodiments described herein, an alarm image frame is generated when an event is detected and associated with an image frame (hereinafter referred to as an "event image frame" in the stream of image frames). The alarm frame is similar to the image frame, but differs in that at least a portion of the alarm frame contains motion relative to the event image frame (i.e., one or more motion vectors are applied to at least a portion of the event image frame when the alarm image frame is generated), or contains color variations relative to the event image frame (e.g., the alarm image frame appears redder compared to the event image frame). The alarm image frame is hard-coded, meaning it is not encoded by an encoder. Further, the hard-coded alarm image frame is generated in software, for example, by an image processing pipeline (IPP). This can also be expressed as the hard-coded alarm image frame being generated by software, for example, executing in an IPP. Various combinations of motion and color variations in the alarm image frame are also possible.

[0038] Alarm image frames are inserted into the image frame stream at the position of event image frames. Therefore, hard-coded alarm image frames can be inserted into the encoded image frame stream at the position of encoded event image frames. More specifically, in one embodiment, alarm image frames are inserted immediately after event image frames. For example, hard-coded alarm image frames can be inserted into the encoded image frame stream in display order after the encoded event image frames. In such embodiments, alarm image frames will be displayed after and additionally for the event image frames. In some embodiments, event image frames are changed to "not displayed," meaning that event image frames are not displayed. In such embodiments, alarm image frames are displayed instead of event image frames. This process can be repeated for several event image frames in the image frame stream, and different motions and / or different color changes (e.g., different intensities) are typically applied to alarm image frames for several seconds. As a result, the video feed on the operator's display exhibits "jitter" and / or "flickering," which increases the likelihood that the operator will detect events that might otherwise be undetectable without the aforementioned operations. Furthermore, after the operator becomes aware of the event, the stream of image frames can be altered so that alarm image frames are removed and the status not displayed is removed from the event image frames. This allows the operator to inspect the event captured by the camera. Various embodiments of the invention will now be described by way of example and with reference to the accompanying drawings.

[0039] System Overview

[0040] Figure 1 A schematic diagram of an exemplary environment 100 in which various embodiments of the present invention may be implemented is shown. (See diagram from...) Figure 1 As can be seen, scene 102, featuring person 104 and tree 106, is captured by camera system 108. It should be noted that the depiction of scene 102 is merely a simplified view for illustrative purposes. Scene 102 can be described in a more general sense as any three-dimensional physical space, and its size and shape are defined by the field of view of the camera recording the scene.

[0041] exist Figure 3The camera system 108, illustrated in more detail, has a lens 110 that captures a scene 102 and projects it onto an image sensor 112. The image sensor 112 captures a series of images that together form a video stream. The image sensor is coupled to an image processing pipeline 302 and an encoder 304, both of which will be described in further detail below. The image processing pipeline 302 and encoder 304 are preferably located inside the camera system 108, but may also be located outside the camera system. For example, in a modular camera system, the lens 110 and image sensor 112 may be included in an image capture module, and the image processing pipeline 302 and encoder 304 may be included in an image processing and encoding module. The image capture module and the image processing and encoding module may be arranged separately from each other and configured to communicate with each other via wireless or wired connections. Furthermore, the image capture module may be movable, while the image processing and encoding module may be fixed. The image processing pipeline 302 acquires signals from the image sensor 112 and performs various types of image processing operations, then encodes the video stream into a format suitable for transmission to an operator over a network. Figure 1 In this process, the encoded video is wirelessly transmitted to the wired network 118 via radio link 116, and finally transmitted to the client 120 connected to the network 118. However, there are of course multiple combinations of wireless and wired transmission modes that can be used.

[0042] Client 120 has a display in which an operator can view the video stream of images from the camera. Typically, client 120 is also connected to a server where the video can be stored and / or further processed. Client 120 is also typically used to control camera 108, for example, by issuing control commands from the operator at client 120. For example, the operator can instruct the camera to zoom in on specific details of scene 102, or instruct the camera to track person 104 as person 104 begins to move away from tree 106. However, there are also cases where the operator does not control the camera, and the camera is stationary and only provides an image stream for the operator to view on client 120.

[0043] Client display and user experience

[0044] Figure 2 This diagram illustrates what the operator can see on the monitor of her client 120. (As shown from...) Figure 2 As can be seen, the display is divided into four sections 204a to 204d, each displaying video feeds from a separate camera. For ease of illustration, Figure 2Only four individual feeds are shown, but it should be noted that there can typically be nine, sixteen, or even more feeds on a single display. Section 204a shows a scene depicting a car, section 204b shows a scene depicting a person looking through a window, and section 204c shows... Figure 1 Scene 102, and section 204d shows a person running past the camera. As explained above, an operator (typically viewing multiple displays, each with multiple sections) can easily miss subtle or fleeting events occurring in one of the sections, such as a person moving slightly towards a window in section 204b. This is especially true if the operator's attention might already be focused on different displays when the event occurs. Before describing in detail how various embodiments of the invention address this problem, a brief description of the camera components for image processing will be provided.

[0045] like Figure 3 As shown, the camera system 108 includes a lens 110 that images the scene 102 onto an image sensor 112. After performing various operations (typically filtering, demosaicing, and color correction), the resulting image is forwarded to an image processing pipeline (IPP) 302. It should be noted that in some embodiments, color correction may be performed within the IPP 302.

[0046] In IPP 302, further processing is performed on the image. This further processing may include: noise filtering (for eliminating spatial and / or temporal noise), distortion correction (for eliminating the effects of, for example, barrel distortion), global and / or local tone mapping (e.g., enabling imaging of scenes containing a wide intensity range), transformation (e.g., rotation), flattening correction (e.g., for eliminating vignetting effects), and applying overlays (e.g., privacy masking, explanatory text, etc.). IPP 302 may also be associated with an analysis engine (not shown) that performs object detection, object recognition, alerts, etc. IPP 302 may also be configured to generate hard-coded alert image frames that include motion increments and / or color changes relative to the event image frame. As previously described, the hard-coded alert image frames can be generated by IPP 302 in software. Therefore, IPP 302 can execute computer program code instructions to generate hard-coded alert image frames. Further, IPP 302 may be configured to generate inter-frame image frames that include motion increments and / or color changes relative to the event image frame from the hard-coded alert image frames. As is known to those skilled in the art, an inter-frame image frame includes: a reference to a reference frame to be used when decoding the inter-frame image frame; a frame number indicating the decoding order of the inter-frame image frames, which will be used by the decoder to decode the inter-frame image frames in the correct order; and an indication of the display order, which will be used by the decoder to display the inter-frame image frames in the correct display order. Therefore, IPP 302 generates hard-coded alarm image frames to include a reference to the event image frame, a frame number, and an indication of the display order.

[0047] Following image IPP 302, the image (i.e., the image to be encoded by the encoder) can be forwarded to encoder 304, where information is encoded according to the encoding protocol and forwarded to receiving client 120 via network 118. Further, as will be described below, for example, hard-coded alarm image frames generated in software by IPP 302 will be forwarded to encoder 304 so as to be inserted into the stream of image frames encoded by encoder 304. It should be noted that in Figure 3 The camera system 108 illustrated also includes many other components such as a processor and memory, which are common in conventional camera systems, and whose purpose and operation are well known to those skilled in the art. For clarity, in Figure 3These components are omitted in the illustrations and descriptions. Many conventional video coding formats exist. Some common video coding formats that work with the various embodiments of the present invention include (only a few examples are given): High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2; Advanced Video Coding (AVC), also known as H.264 and MPEG-4 Part 10; Universal Video Coding (VVC), also known as H.266, MPEG-I Part 3, and Future Video Coding (FVC); VP9, ​​VP10, and AOMediaVideo1 (AV1). These video coding formats specify temporal video compression implemented in terms of intra-frame and inter-frame image frames. Intra-frame image frames may also be referred to as intra-frame frames, I-frames, or I-frames, and inter-frame image frames may also be referred to as inter-frame frames and can be predicted image frames (e.g., forward predicted image frames) referred to as P-frames or bidirectional predicted image frames referred to as B-frames. I-frames can be described as image frames encoded using only the information in the image frames to be encoded. Furthermore, an I-frame is calculated based on all the image data captured for the image frame to be encoded. Therefore, an I-frame is sometimes also called a complete image frame.

[0048] P-frames can be based on information from previously encoded image frames and information from the currently encoded image frame. B-frames can be based on information from previously encoded and optionally later encoded image frames and information from the currently encoded image frame. In other words, inter-frame image frames can be described as utilizing temporal redundancy information from previous (and optionally later) image frames. Encoders implementing this type of codec (compression standard) typically generate I-frames followed by a predetermined number of inter-frame image frames (e.g., P-frames and / or B-frames), and then new I-frames followed by the same number of inter-frame image frames. The length of this sequence of I-frames followed by multiple inter-frame image frames is usually called the group of pictures (GOP) length. For some compression standards such as H.265, the GOP length can be adjusted during encoding.

[0049] Figure 4 A method for processing a stream of image frames captured by a camera, according to one embodiment, is shown. (As in...) Figure 4 As can be seen from the above reference, in step 402, the method begins as described above. Figure 3The description describes a stream of image frames processed in a conventional manner. In step 404, continuous monitoring is present during the normal processing in step 402 to detect whether an event has occurred. As mentioned above, event detection itself is separate from the scope of this invention. However, event detection can always be associated with one or more event image frames. For example, in one embodiment, an event can be detected by analyzing the contents of image frames captured by camera system 108. In another embodiment, an event can be associated with image frames based on time. For example, by knowing when a sensor was triggered (e.g., a door or window was opened), it can be determined which image frames were captured at that exact time. Regardless of how an event is detected, it always corresponds to one or more image frames, referred to herein as event image frames. It should be noted that event image frames are not necessarily image frames captured at the exact time the event occurred, but rather events can be associated with event image frames captured within a specific time frame after the event occurred. For clarity, the examples described herein will generally refer to a single event image frame, but it should be understood that these techniques are applicable to several event image frames, which may be continuous or non-continuous and may be captured over the entire duration of the event or only a portion of its duration. If no event occurs, normal processing continues. However, if an event is detected to have occurred in step 404, an alarm image frame is generated in step 406. The alarm image frame corresponds to the event image frame but has some different characteristics, as will now be described.

[0050] In one embodiment, an alarm image frame is a P-frame that has a predefined leftward or rightward movement relative to at least a portion of an event image frame. Typically, this movement is implemented using motion vectors, as is well known to those skilled in the art. Depending on the specific implementation, both the amount and direction of the movement can be configurable or predefined. For example, encoder 304 can support both horizontal and vertical movement, and encoder 304 also allows any kind of diagonal movement. As previously mentioned, hard-coded alarm image frames are generated in software, for example, by IPP 302. However, for a decoder that decodes a hard-coded alarm image frame, the hard-coded alarm image frame should “look” the same as that encoded by encoder 304. Therefore, for the decoder, there is no difference between an image frame encoded by encoder 304 and a hard-coded alarm image frame generated in software, for example, by IPP 302. Thus, hard-coded alarm image frames can also include motion increments relative to the event image, providing both horizontal and vertical movement. In the presence of several event image frames (which is common), corresponding alarm image frames can be generated such that they include movement in alternating directions. This will be described in further detail below, but essentially, the image stream, in which alternating alarm image frames include alternating motion, enables a "jittering" effect when displayed on monitor 120, which helps to attract the operator's attention. It should be noted that the motion vector can be a global motion vector, causing the entire alarm image frame to move a certain amount in a certain direction relative to the event image frame. Alternatively, the motion vector can be applied only to a portion of the event image frame, such as a specific object of interest, causing only that portion of the alarm image frame to move relative to the event image frame.

[0051] In another embodiment, the alarm image frame is a P-frame that has a predefined color variation relative to at least a portion of the event image frame. For example, the alarm image frame may have a more or less color representation relative to the event frame (e.g., more or less red). Of course, the colors do not have to be the same in different alarm image frames. For example, one alarm image frame may have a redder hue, while a subsequent alarm image frame may have a bluer hue. However, given that people are accustomed to viewing red as "danger" or "alarm," having different intensities of red in alarm image frames is often a good choice. Similar to motion vectors, color variations can be applied to the entire alarm image frame or a portion of it. Furthermore, a portion of the image frame does not have to be related to the content of the event image frame, but can be, for example, the boundary of a highlighted area surrounding the alarm image frame. Such examples are... Figure 5 As shown, an event occurred in portion 204c of display 120.

[0052] Once the alarm image frame (i.e., the hard-coded alarm image frame) is generated, in step 408, it is inserted into the stream of encoded image frames after the event image frame (typically directly after the event image frame), and the corresponding event image frame is marked as a non-display image frame. By essentially "replacing" the event image frame with the alarm image frame, the original frame rate is maintained, except for jitter and / or color variations, and the content in the image stream still appears normal to the operator. Furthermore, when the operator is already aware of the event, the alarm image frame can be removed, and the non-display feature of the event image frame can also be removed, allowing the operator to determine whether the event that triggered the alarm requires any attention.

[0053] It should be noted that since events typically have a certain duration, there may be many event image frames associated with the event. In some embodiments, alarm image frames are generated for each event image frame, while in other embodiments, alarm image frames are generated only for a portion of the event image frames. Those skilled in the art can determine exactly how many event image frames should be used to generate alarm image frames based on the current situation. However, as a general guideline, it is recommended that regardless of how many alarm image frames are generated, they include movement in alternating directions and / or varying degrees of color intensity variation, as this will make these changes more noticeable to the operator. Furthermore, the way event image frames are selected allows for the configuration of various shake modes. For example, there could be three “slight, short shakes,” followed by a pause in the shake and the display of the event image frames, and then a “long, large shake.” In many ways, this is similar to what a mobile phone can do in vibration mode.

[0054] In some embodiments, "jitter" can continue after the event has ended and until the operator acknowledges the event. For example, there are situations where the event is very short and the operator might still miss it if the jitter only occurs during the event. Therefore, to increase the likelihood that the operator will detect the event, the system can continue generating alarm image frames until the operator acknowledges the event. This can be achieved, for example, by setting multiple "normal" image frames following the event frame to "not display" and then adding the corresponding number of generated alarm frames to the stream.

[0055] In step 410, once a stream of image frames containing the alarm image frames has been generated, it is determined whether there are any more image frames to process. If so, normal image processing continues in step 402, as described above. If there are no more image frames to process, for example, if the camera is off or in sleep mode, method 400 ends.

[0056] To further illustrate the versatility of the invention as described herein, consider the following examples.

[0057] Event trigger.

[0058] The system has been configured to alert the operator by flashing a red alarm, which repeats in red from 0% to 100% mixed with the original image, for example, 0% for event image frame #1, 50% for event image frame #2, 100% for event image frame #3, 50% for event image frame #4, 0% for event image frame #5, 50% for event image frame #6, and so on.

[0059] For certain event image frames, particularly event image frames #2, #3, #4, #6, etc., alarm image frames are generated and added to the image frame stream, while event image frames #2, #3, #4, #6, etc., are marked as "not displayed". However, for event image frames #1, #5, etc. (i.e., alarm image frames using 0% mixing will be the same as the corresponding event image frames), it is not necessary to generate alarm image frames (nor is it necessary to mark these image frames as "not displayed" image frames).

[0060] In the above example, event image frames #1 and #5 will be encoded normally, while event image frames #2, #3, #4, and #6 will be encoded by encoder 304 as not to be displayed. A first hard-coded alarm image frame A1 will be generated, for example, by IPP 302 in software with a 50% red blend relative to event image frame #2, and this first hard-coded alarm image frame A1 will be inserted by encoder 304 into the stream of encoded image frames after event image frame #2 in display order. Similarly, a second hard-coded alarm image frame A2 will be generated, for example, by IPP 302 with a 100% red blend relative to event image frame #3, and this second hard-coded alarm image frame A2 will be inserted by encoder 304 into the stream of encoded image frames after event image frame #3 in display order. Furthermore, the third hard-coded alarm image frame A3 will be generated, for example, by IPP 302 in software using a 50% red blend relative to event image frame #4, and this third hard-coded alarm image frame A3 will be inserted by encoder 304 into the stream of encoded image frames after event image frame #4 in display order. Correspondingly, the fourth hard-coded alarm image frame A4 will be generated, for example, by IPP 302 in software using a 50% red blend relative to event image frame #6, and this fourth hard-coded alarm image frame A4 will be inserted by encoder 304 into the stream of encoded image frames after event image frame #6 in display order. Therefore, the image frame stream will include the following image frames in display order: event image frame #1, event image frame #2 (not displayed), first hard-coded alarm image frame A1 (50% red relative to #2), event image frame #3 (not displayed), second hard-coded alarm image frame A2 (100% red relative to #3), event image frame #4 (not displayed), third hard-coded alarm image frame A3 (50% red relative to #4), event image frame #5, event image frame #6 (not displayed), and fourth hard-coded alarm image frame A4 (50% red relative to #6). As those skilled in the art will understand, inserting image frames into the image frame stream may require updating the frame numbers of one or more image frames, possibly referencing one or more reference frames and indications of display order to ensure correct decoding of image frames and the correct decoding order, and to ensure the correct display order when displaying the image frame stream.

[0061] This example demonstrates that it is not necessary to generate an alert image frame for every event image frame in the stream of image frames; generating an alert image frame for only a subset of the event image frames may suffice.

[0062] In one embodiment, there may be two separate "tracks," one containing a stream of raw image frames and the other containing only alarm image frames. If the first track is set to "not displayed," only the stream of alarm image frames will be shown to the operator. As a result, the operator's attention will be drawn to the "shaky still image" captured in the second track, and the operator can then switch back to the first track and remove the undisplayed feature to see what the camera actually captured.

[0063] Conclusion

[0064] It should be noted that while the examples above focus on using P-frames, the same general principles of the invention also apply to B-frames, which can be both forward-referenced and backward-referenced within a GOP. However, B-frames typically have higher memory requirements than P-frames, so using P-frames is preferred in most cases.

[0065] Furthermore, while the examples above have been described as discrete embodiments in which movement or color change occurs within the alarm image frame, nothing prevents these embodiments from being combined. For example, jitter and flickering effects can be created, which can further increase the likelihood that an operator will quickly notice a video feed containing the event. In some embodiments, other effects can be applied. For example, image distortion or “bending” can occur instead of (or in combination with) jitter and flickering. Furthermore, in some implementations, the type of alarm can be changed if the alarm is not acknowledged by the operator within a certain amount of time. For example, if a “jitter” alarm is not acknowledged, it may be changed to a flickering alarm, etc.

[0066] It should also be noted that in some implementations, various types of auditory alarms (warning sounds or alarms, etc.) may be associated with visual alarms displayed on the monitor.

[0067] The systems (e.g., image processing pipelines and / or encoders) and methods disclosed herein can be implemented as software, firmware, hardware, or a combination thereof. In a hardware implementation, the task division among the functional units or components described above does not necessarily correspond to the division of physical units; rather, a physical component can perform multiple functions, and a task can be completed collaboratively by several physical components.

[0068] Specific components or all components may be implemented as software executed by a digital signal processor or microprocessor, or as hardware or application-specific integrated circuits. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disk (DVD) or other optical disc storage, cassette tape, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.

[0069] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, depending on the function involved, two blocks shown consecutively may actually be executed substantially simultaneously, or sometimes these blocks may be executed in reverse order. It will also be noted that each block illustrated in the block diagrams and / or flowcharts, and combinations of blocks illustrated in the block diagrams and / or flowcharts, may be implemented by systems based on dedicated hardware that perform the specified functions or actions or execute combinations of dedicated hardware and computer instructions.

[0070] It will be understood that those skilled in the art can modify the above embodiments in various ways while still utilizing the advantages of the invention as shown in the above embodiments. Therefore, the invention should not be limited to the illustrated embodiments but should be defined only by the appended claims. Furthermore, as those skilled in the art will understand, the illustrated embodiments can be combined.

Claims

1. A method of processing a stream of image frames in a camera system, comprising: in response to detecting an event, generating a hard-coded alert image frame, wherein the hard-coded alert image frame comprises a motion delta and / or a color change relative to an event image frame of the stream of image frames, wherein the event image frame is captured at a time corresponding to a time when the event occurred, and wherein the hard-coded alert image frame is generated in software to produce a desired change in the video stream when displayed to an operator; generating a stream of encoded image frames; and inserting the hard-coded alert image frame in the stream of encoded image frames after the encoded event image frame in a display order, wherein the hard-coded alert image frame is an interframe image frame.

2. The method of claim 1, further comprising: encoding the event image frame as a non-displayed image frame.

3. The method of claim 1, wherein, the event is an event detected in an event image frame of a stream of image frames or is an external event.

4. The method of claim 1, wherein, the motion delta comprises a motion in a horizontal direction, a vertical direction, and any combination thereof relative to the event image frame.

5. The method of claim 1, wherein, the motion delta has a configurable size or a predefined size.

6. The method of claim 1, wherein, the motion delta is applied to only a portion of the alert image frame.

7. The method of claim 1, wherein, the color change comprises one or more of a representation of more colors relative to the event image frame, a representation of less colors relative to the event image frame, and a representation of changed colors relative to the event image frame.

8. The method of claim 1, wherein, the color change is applied to only a portion of the alert image frame.

9. The method of claim 2, further comprising: in response to an input by an operator after confirming the event, removing the non-displayed status of the event image frame and modifying a status of the alert image frame to be non-displayed to enable the operator to view the event captured by the camera system (108).

10. The method of claim 1, wherein, the alert image frame is one of a forward-predicted frame P-frame containing a motion delta relative to the event image frame and a bi-directional frame B-frame containing a motion delta relative to the event image frame.

11. The method of claim 1, wherein, the event image frame is a reference frame of a group of pictures GOP.

12. A camera system, comprising: a lens and an image sensor configured to capture a stream of image frames; an image processing pipeline configured to: in response to detecting an event, generate a hard-coded alert image frame, wherein the hard-coded alert image frame comprises a motion delta and / or a color change relative to an event image frame of the stream of image frames, wherein the event image frame is captured at a time corresponding to a time when the event occurred, and wherein the hard-coded alert image frame is generated in software to produce a desired change in the video stream when displayed to an operator; and an encoder configured to: generate a stream of encoded image frames; and insert the hard-coded alert image frame in the stream of encoded image frames after the encoded event image frame in a display order, wherein the hard-coded alert image frame is an interframe image frame.

13. A non-transitory computer readable storage medium having program instructions included thereon, the program instructions executed by a processor to perform a method comprising: generating a hard-coded alert image frame in response to detecting an event, wherein the hard-coded alert image frame includes a motion delta and / or a color change relative to an event image frame of a stream of the image frames, wherein the event image frame is captured at a time corresponding to a time when the event occurred, and wherein the hard-coded alert image frame is generated in software to produce a desired change in the video stream when displayed to an operator; and generating a stream of encoded image frames; and inserting the hard-coded alert image frame after the encoded event image frame in the stream of encoded image frames in a display order, wherein the hard-coded alert image frame is an interframe image frame.

Images