Warning generation based on event detection within video footage

Abstract

To provide a method, a system, and a computer program product that generate an alert to an operator in response to detecting an event in a video material.SOLUTION: The method is for processing a stream of image frames captured by a camera system. A hard-coded alert image frame is generated in response to detecting an event. The hard-coded alert image frame includes motion deltas and / or color changes with respect to an event image frame. The method further generates a stream of encoded image frames. The hard-coded alert image frames are inserted into the stream in a display order after the encoded event image frames.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a video surveillance system, and more particularly to generating a warning to an operator in response to detecting an event in video material.

Background Art

[0002] In many surveillance systems, video or other data from a number of cameras and / or other sensors are managed in an operations control room and displayed on monitor screens. Typically, there are several screens, such as 3 to 6 screens, in the operations control room. Each screen displays several video materials, for example, 4×4 video materials, which are monitored by an operator. As a result, the operator has to pay attention to 48 (3×16) to 96 (6×16) video materials simultaneously in order to detect an event in a single video material.

[0003] Events can include, for some examples, the movement of an object such as a vehicle, an intruder into a restricted area, a detected face, and a crowded area. However, due to the large amount of video material, there is a risk that an event in one of the video materials cannot be detected, especially when the event is subtle or occurs very quickly while the operator's attention is temporarily diverted from the specific video material containing the event. Therefore, there is a need to direct the operator's attention to a certain video material when there is an event in that video material.

[0004] The video surveillance system is described in International Publication No. 2006 / 006081. The video surveillance system consists of smart cameras, a server, and clients connected via an IP network in a wired or wireless configuration. The system has been designed to protect the privacy of the people and goods being monitored. The operation of the smart cameras is optimized to provide a balance between the perceived visual quality of the decoded video and power consumption. The server receives, stores, manages, and transmits video sequences over wired and wireless channels to various clients and users with different device capabilities, channel characteristics, and preferences. The use of seamless and scalable coding of video sequences eliminates the need for transcoding operations at any point in the system.

[0005] U.S. Patent Application Publication 2003 / 0122667 describes a system and method for enhancing security at self-checkout stations. The system includes a security agent application that runs on terminals at several self-checkout stations. The security agent software generates event messages about security events occurring at the station and sends them to a server. The server sends priority event messages as alert messages to a security controller. The security controller is coupled to security cameras, image data storage devices, and image data display devices and generates control messages for these devices in response to received alert messages. Control messages for security cameras cause the cameras to zoom, focus, tilt, or pan in relation to events occurring at the station. The controller may insert visual alert instructions into the video stream of the monitored camera, or an audible tone into the audio of the video stream, to draw the attention of security personnel to the display of ongoing security events at the station. [Overview of the project]

[0006] According to a first aspect, the present invention relates to a method in a camera system for processing a stream of image frames. The method is: • In response to detecting an event, generate a hardcoded warning image frame, the hardcoded warning image frame being an inter-image frame containing motion delta and / or color changes relative to the event image frame, and the hardcoded warning image frame being generated by software to produce a desired change in the video stream when displayed to the operator, and This involves generating a stream of encoded image frames, in which hardcoded warning image frames are inserted into the stream of encoded image frames in the order they appear, after the encoded event image frames.

[0007] This method makes it easier for operators to be alerted regarding video material in monitoring situations where suspicious events may occur. The method can be applied to any type of event, rather than being limited by the type of event and / or detection mechanism. The event image frame used herein is an image frame correlated with the event. The event image frame can be an image frame containing the actual event, or an image frame captured at or near the time the event occurred. Furthermore, when used in this disclosure, the representation, hardcoded warning image frame should be understood as a warning image frame generated by software to create a desired change in the video when displayed to the operator. Therefore, it should be understood that the hardcoded warning image frame is not created by the encoder by encoding the image data. Because the warning image frame is hardcoded, it does not need to be encoded by the encoder, but can be directly inserted into the image stream output from the encoder. The desired change in the video when displayed to the operator can be a shaking motion and / or a color change in the displayed video to alert the operator. The type of warning, i.e., "shaking" and / or color change, is configurable so that the most appropriate type of warning can be selected considering the situation at hand.

[0008] As a person skilled in the art will understand, an encoded inter-image frame includes 1) a reference frame, i.e., a reference to an image used in inter-prediction when the encoded inter-image frame is decoded by a decoder; 2) a frame number that allows the decoder to decode in the correct decoding order; and 3) a display order instruction that allows the decoder to display the decoded inter-image frame at the correct temporal position in the decoded video stream. Thus, a person skilled in the art will understand that a hardcoded warning image frame, which is an inter-image frame, includes 1) a reference to an event image frame used when decoding a hardcoded warning image frame that includes motion delta and / or color changes relative to an event image frame; 2) a frame number that indicates in what decoding order the hardcoded warning image frame should be decoded; and 3) a display order instruction that allows the decoder to display the decoded hardcoded warning image frame in the correct display in order to create a desired change in the video stream when displayed to the operator. Sometimes, in this disclosure, hardcoded warning image frames are simply referred to as warning image frames.

[0009] According to one embodiment, the method further includes encoding the event image frame as a no-display frame. While it is possible to warn the operator without encoding the event image frame as a no-display frame, the visual effect is often better when the event image frame is encoded as a no-display frame. Furthermore, the original display frame rate can be maintained by encoding the event image frame as a "no-display" image frame and inserting a warning image frame after each event image frame (i.e., in the display order, even if the event image frame is a no-display image frame and therefore not displayed, the warning image frame should be inserted after the event image frame as if the event image frame were displayed), thus giving the video on the operator's display a natural look. In this context, "no-display" means that the image frame is not displayed to the user. The H.265 encoding standard (and other newer encoding standards such as Google's VP10, for example) allows frames to be tagged as "no-display". For example, in H.265, tagging can be performed by setting pic_output_flag to false in the slice header or by setting no_display_flag to true in the SEI header.

[0010] According to one embodiment, an event is an event detected within an event image frame in a stream of image frames or an external event. That is, as stated above, the method is not limited to any particular type of event detection. For example, an event can be detected by, for example, a camera system by analyzing the content of an image frame, but an event can also be triggered by an external sensor, such as a door being opened, and correlated with an event image frame captured at the time (or very close to) when the sensor is triggered. This greatly increases the situations in which the method according to the present invention can be used.

[0011] According to one embodiment, both generating steps are repeated for several event image frames. Even if the method described herein is used only for a single event image frame, it will still provide an improvement over existing techniques in drawing the operator's attention to an event. However, by repeating the generating steps of the method for several event image frames, it is possible to achieve a “vibration” or “flashing” appearance in the image stream, which makes it easier for the operator to be alerted that an event has occurred in particular video material. The vibration / flashing can continue for a predetermined period of time or until the operator acknowledges the warning.

[0012] According to one embodiment, the motion delta includes motion relative to the event image frame in the horizontal, vertical, and any combination thereof. Most encoders are capable of applying both horizontal and vertical motion. Therefore, it should be understood that, similarly, a software-generated warning image frame, i.e., a hardcoded warning image frame, can also contain a motion delta that includes both horizontal and vertical motion, for example, through an image processing pipeline. This makes it possible to move (and thus "shake") an image in an image stream in essentially any pattern desired by the user. In some cases, these patterns can also correlate with certain types of events. For example, a suspicious person entering the scene might cause a "small vertical shaking," while someone attempting to break into a building might cause a "bright red flashing" or similar.

[0013] According to one embodiment, the motion delta has a configurable or predetermined size. The ability to select a predetermined or configurable size for the motion increases the versatility of the present invention and makes it adaptable to many different use cases. For example, the size of the motion delta may relate to various characteristics of the event, such as type, intensity, importance, or accuracy, to name a few examples.

[0014] According to one embodiment, the motion delta is applied to only a portion of the warning image frame. That is, in some cases, it may not be necessary to apply motion to the entire warning image frame. For example, if an object in a scene is placed against a uniformly colored wall background, there will be few or no points that would "vibrate" the entire wall, but it would suffice to vibrate only the object, thereby conserving computational resources while still allowing the operator to be warned.

[0015] According to one embodiment, the color change includes one or more of the following: a more colorful representation compared to the event image frame, a less colorful representation compared to the event image frame, and a representation in which the colors change compared to the event image frame. This allows any color configuration to occur within the warning image frame, and can create various effects such as slightly stronger flashing, alternating color flashing, or converting the entire warning image frame to a uniform color.

[0016] According to one embodiment, the color change is applied to only a portion of the warning image frame. As described above, in some cases it may be sufficient to apply the color change only to a specific object of interest, which would highlight the object while leaving the rest of the warning image frame unchanged. This could be useful, for example, in a facial recognition scenario where only certain individuals in a group of people may be of interest to the operator. Furthermore, the color change does not necessarily have to be applied to an object; for example, a specific portion of the warning image frame, such as the border around the warning image frame, could be colored, or the warning image frame could be striped or have some kind of other pattern, which could be particularly useful when a particular warning pattern is linked to a particular event type.

[0017] According to one embodiment, the method further includes, in response to input from the operator, removing the hidden state of event image frames and changing the state of warning image frames to hidden in order to allow the operator to observe events captured by the camera system. That is, after the operator has been warned and acknowledged the warning, she can re-examine the original images captured by the camera to determine what the event was and whether any action needs to be taken.

[0018] According to one embodiment, the warning image frame is either a forward-predicting frame (P-frame) containing a motion delta relative to the event image frame, or a bidirectional frame (B-frame) containing a motion delta relative to the event image frame. P-frames and B-frames are standard concepts well known in the context of image processing, thereby facilitating the integration of the invention described herein with existing and future monitoring systems.

[0019] According to one embodiment, a hardcoded warning image frame is generated based on a warning image frame generated outside the camera system. Therefore, the generation of a hardcoded warning image frame may include generating a hardcoded warning image frame based on a warning image frame generated outside the camera system. This reduces the computational power required by the camera and makes the present invention applicable to a large number of devices. For example, a warning image frame can be generated by an external application that has no control over the camera image processing pipeline or a video management system (VMS) event handling system. The external application can be triggered by an external device such as a microphone, radar, passive infrared (PIR) sensor, motion sensor, door sensor, window sensor, etc. The warning image frame generated by the external application includes a predetermined or specified motion delta and / or color change. Furthermore, the warning image frame can be input into an image processing pipeline that can generate a hardcoded warning image frame based on an externally generated warning image frame. Since hardcoded warning image frames are generated relative to event image frames, it should be understood that the image processing pipeline generates the hardcoded warning image frames as inter-image frames that include motion delta and / or color changes from the input warning image frames as well as references to the event image frames. As stated above and known to those skilled in the art, the inter-image frames also include frame numbering and indication of display order.

[0020] According to one embodiment, the event image frame is a reference frame for a Group of Pictures (GOP). A GOP is a well-known and standard concept in the context of image processing, thereby facilitating the integration of the invention described herein with existing and future monitoring systems.

[0021] According to a second aspect, the present invention relates to a camera system. The camera system includes a lens, an image sensor, an image processing pipeline, and an encoder. The lens and image sensor are configured to capture a stream of image frames. The image processing pipeline is configured to generate a hardcoded warning image frame in response to detecting an event, the hardcoded warning image frame being an inter-image frame and including a motion delta and / or color change with respect to the event image frame, the hardcoded warning image frame being generated in software to produce a desired change in the video stream when displayed to an operator. The encoder is configured to generate a stream of encoded image frames, the hardcoded warning image frame being inserted into the stream of encoded image frames in display order after the encoded event image frame.

[0022] The advantages of a system correspond to the advantages of a method, and may also vary.

[0023] According to a third aspect, the present invention relates to a computer program product for processing a stream of image frames captured by a camera. The computer program includes instructions, and the instructions are: · In response to detecting an event, generating a hard-coded warning image frame, wherein the hard-coded warning image frame is an inter-image frame and includes motion deltas and / or color changes with respect to an event image frame, and the hard-coded warning image frame, when displayed to an operator, is generated by software to create a desired change in the video stream; and, · Generating a stream of encoded image frames, wherein the hard-coded warning image frame is inserted into the stream of encoded image frames after the encoded event image frame. corresponds to

[0024] The computer program includes advantages corresponding to the advantages of the method and may vary similarly.

[0025] Details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features and advantages of the invention will be apparent from the description and drawings, and from the claims.

Brief Description of the Drawings

[0026] [Figure 1] FIG. 1 is a schematic diagram showing an exemplary environment 100 in which various methods and systems of the invention can be applied. [Figure 2] FIG. 2 is a schematic illustration of a display showing four video materials according to one embodiment. [Figure 3] FIG. 3 is a block diagram showing a detailed view of the camera system 108 shown in FIG. 1 according to one embodiment. [Figure 4] FIG. 4 is a flowchart showing a method for processing a stream of image frames according to one embodiment. [Figure 5] FIG. 5 is a schematic illustration of a display showing four video materials according to one embodiment, with one video material modified to draw an operator's attention.

[0027] Similar reference numerals in various drawings indicate similar elements. [Modes for carrying out the invention]

[0028] As described above, one objective of the various embodiments of the present invention is to provide an improved technique for directing the operator's attention to video footage when an event is present within that video footage. Strictly speaking, what constitutes an "event" is outside the scope of the present invention and can be determined separately. However, as stated in the introduction to this specification, examples of events include the movement of objects such as vehicles, intruders entering restricted areas, detected faces, crowded areas, etc. These can all be considered "visual" events. However, there may also be events such as sounds, something breaking, or something malfunctioning, which may not necessarily be visual but can also be handled according to the techniques described below. Various embodiments of the present invention relate to what happens after an event is detected.

[0029] According to various embodiments described herein, when an event is detected and associated with an image frame in a stream of image frames, hereafter referred to as an "event image frame," a warning image frame is generated. A warning frame is similar to an image frame, but differs in that at least a portion of the warning frame includes motion with respect to the event image frame (i.e., one or more motion vectors are applied to at least a portion of the event image frame when generating the warning image frame), or includes color changes with respect to the event image frame (e.g., the warning image frame appears redder, for example, compared to the event image frame). Warning image frames are hardcoded, meaning they are not encoded by an encoder. Furthermore, hardcoded warning image frames are generated in software, for example, by an image processing pipeline (IPP). This can also be expressed, for example, as a hardcoded image frame produced by software running in an IPP. Various combinations of motion and color changes are also possible within a warning image frame.

[0030] Warning image frames are inserted into the stream of image frames at the location of event image frames. Therefore, hardcoded warning image frames can be inserted into the scene of encoded image frames at the location of encoded event image frames. More specifically, in one embodiment, warning image frames are inserted immediately after event image frames. For example, hardcoded warning image frames can be inserted into the scene of encoded image frames in display order after encoded event image frames. In such embodiments, warning image frames will be displayed after and in addition to event image frames. In some embodiments, event image frames are changed to "hidden," meaning that event image frames are not displayed. In such embodiments, warning image frames will be displayed in place of event image frames. This process can be repeated for several event image frames in the stream of image frames, often for several seconds, with different motion and / or different color changes (e.g., different intensities) applied to the warning image frames. As a result, the video material on the operator's display appears to "shake" and / or "flash," increasing the likelihood that the operator will detect events that might otherwise go undetected. Furthermore, the sequence of image frames can be modified after the operator acknowledges an event, such that warning image frames are removed and hidden states are removed from event image frames, allowing the operator to examine the events captured by the camera. Various embodiments of the present invention are described here by example and with reference to the drawings.

[0031] System Overview Figure 1 shows a schematic diagram of an exemplary environment 100 in which various embodiments of the present invention can be implemented. As can be seen in Figure 1, a scene 102 having a person 104 and a tree 106 is captured by a camera system 108. It should be noted that the depiction of scene 102 is merely a simplified view for illustrative purposes. Scene 102 can be described in a more general sense as an arbitrary three-dimensional physical space, whose size and shape are defined by the field of vision of the camera recording the scene.

[0032] The camera system 108, shown in detail in Figure 3, has a lens 110 that captures a scene 102 and projects it onto an image sensor 112. The image sensor 112 captures a series of images that together form a video stream. The image sensor is coupled to an image processing pipeline 302 and an encoder 304, both of which are described in more detail below. The image processing pipeline 302 and encoder 304 are preferably located inside the camera system 108, but can also be located outside the camera system. For example, in a modular camera system, the lens 110 and image sensor 112 can be contained within an image acquisition module, and the image processing pipeline 302 and encoder 304 can be contained within an image processing and encoding module. The image acquisition module and the image processing and encoding module can be located apart from each other and communicate with each other via wireless or wired connections. Furthermore, the image acquisition module can be movable, while the image processing and encoding module can be fixed. The image processing pipeline 302 takes a signal from the image sensor 112 and performs various types of image processing operations before encoding the video stream into a format suitable for transmission to an operator over the network. In Figure 1, the encoded video is wirelessly transmitted via the wireless link 116 to the wired network 118, and ultimately to the client 120 connected to the network 118, although of course there are many combinations of wireless and wired transmission models that can be used.

[0033] Client 120 has a display screen from which an operator can observe the image video stream from the camera. Typically, client 120 is also connected to a server that can store and / or further process the video. Often, client 120 is also used to control camera 108, for example, by the operator issuing control commands on client 120. For example, the operator may instruct the camera to zoom in on a specific detail in scene 102, or to track person 104 if person 104 begins to move away from tree 106. However, there are also situations in which the camera is fixed and only provides an image stream for the operator to observe on client 120, rather than the operator controlling the camera.

[0034] Client display and operator experience Figure 2 provides a schematic view of what an operator can see on the display of the operator's client 120. As can be seen in Figure 2, the display is divided into four sections 204a–204d, each showing video material from a separate camera. For ease of illustration, only four separate pieces of material are shown in Figure 2, but it should be noted that there are often nine, sixteen, or even more pieces of material on a single display. A scene depicting a car is shown in section 204a, a scene depicting a person looking through a window is shown in section 204b, scene 102 in Figure 1 is shown in section 204c, and a person running past the camera is shown in section 204d. As described above, typically, an operator looking at several displays, each having multiple sections, may easily miss a subtle or short-duration event occurring in one of the sections, for example, a person in section 204b slightly closer to the window. This is especially true if the operator's attention is focused on a different display while the event is occurring. Before specifically describing how various embodiments of the present invention address this problem, a brief description of image processing of camera components is provided.

[0035] As shown in Figure 3, the camera system 108 includes a lens 110 that images the scene 102 onto the image sensor 112. After performing various operations, typically filtering, demosaicing, and color correction, the resulting image is transferred to the image processing pipeline (IPP) 302. It should be noted that in some embodiments, color correction can be performed in the IPP 302.

[0036] Further processing is performed on the image in IPP302. Such further processing may include noise filtering (to eliminate spatial and / or temporal noise), distortion correction (e.g., to eliminate the effects of barrel distortion), global and / or local tone mapping (e.g., to enable imaging of scenes with a wide range of intensities), transformation (e.g., rotation), flat-field correction (e.g., to remove the effects of vignetting), and the application of overlays (e.g., privacy masks, descriptive text, etc.). IPP302 can also be associated with an analysis engine (not shown) that performs object detection, object recognition, alarms, etc. IPP302 can also be configured to generate hardcoded warning image frames that include motion delta and / or color changes for event image frames. As mentioned above, hardcoded warning image frames can be generated in software by IPP302. Therefore, IPP302 can execute computer program code instructions to generate hardcoded warning image frames. Furthermore, the IPP302 can be configured to generate a hardcoded warning image frame as an inter-image frame containing motion delta and / or color changes relative to the event image frame. As is known to those skilled in the art, an inter-image frame includes a reference to a reference image used when decoding the inter-image frame, a frame number indicating the decoding order of the inter-image frames used by the decoder to decode the inter-image frames in the correct decoding order, and a display order instruction used by the decoder to display the inter-image frames in the correct display order. Thus, the IPP302 generates a hardcoded warning image frame containing a reference to the event image frame, a frame number, and a display order instruction.

[0037] Following IPP302, the image, i.e., the image to be encoded by the encoder, can be transferred to encoder304, where the information is encoded according to the encoding protocol and transferred to receiving client120 via network118. Furthermore, as described below, for example, hardcoded warning image frames generated by software by IPP302 will be transferred to encoder304 to be inserted into the stream of image frames to be encoded by encoder304. It should be noted that the camera system 108 shown in Figure 3 also includes many other components, such as processors and memory, which are common in conventional camera systems and whose purposes and operations are well known to those skilled in the art. Such components are omitted from the illustration and description in Figure 3 for clarity. Several conventional video encoding formats exist. Some common video encoding formats that work with various embodiments of the present invention include, to name a few, High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2; Advanced Video Coding (AVC), also known as H.264 and MPEG-4 Part 10; Versatile Video Coding (VVC), also known as H.266, MPEG-I Part 3, and Future Video Coding (FVC); VP9, ​​VP10, and AOMedia Video 1 (AV1). These video encoding formats define temporal video compression implemented with respect to intra-image frames and inter-image frames. Intra-image frames may also be called intra frames, I-image frames, or I-frames, and inter-image frames may also be called inter-frames, and can be predictive, for example, forward-predictive image frames, so-called P-frames, or bidirectional predictive image frames, so-called B-frames.An I-frame can be described as an image frame encoded using only the information within the image frame being encoded. Furthermore, an I-frame is calculated from all the image data captured for the image frame being encoded. Therefore, an I-frame is sometimes also called a whole image frame.

[0038] A P-frame can be based on information from previously encoded image frames and information from the currently encoded image frame. A B-frame can be based on information from previously encoded, and optionally, later encoded, image frames and information from the currently encoded image frame. That is, an inter-image frame can be described as utilizing temporally redundant information in previous (and optionally later) image frames. Encoders implementing this type of codec (compression standard) typically generate an I-image frame, followed by a predetermined number of inter-image frames, e.g., P and / or B-frames, and then generate a new I-image frame, followed by the same number of inter-image frames. The length of this sequence of I-frames followed by several inter-image frames is often called the group of pictures (GOP) length. For some compression standards, such as H.265, the GOP length can be adjusted during encoding.

[0039] Figure 4 shows a method for processing a stream of image frames captured by a camera, according to one embodiment. As can be seen in Figure 4, the method begins by processing the stream of image frames in a conventional manner, as described above with reference to Figure 3, in step 402. During the normal processing of step 402, there is continuous monitoring to detect whether or not an event has occurred, in step 404. As stated above, event detection itself is outside the scope of the present invention. However, event detection can always be correlated with one or more event image frames. For example, in one embodiment, an event can be detected by analyzing the content of an image frame captured, for example, by the camera system 108. In another embodiment, an event can be correlated with an image frame based on time. For example, by knowing when a sensor was triggered (e.g., by opening a door or window), it is possible to determine which image frame(s) were captured at that exact time. Regardless of how an event is detected, an event will always correspond to one or more image frames, which will be referred to herein as event image frames. It should be noted that an event image frame does not necessarily have to be an image frame captured at the exact time the event occurred; instead, the event can be associated with an event image frame captured within a certain time frame after the event occurred. For clarity, the examples described herein generally refer to a single event image frame, but it should be recognized that these techniques are applicable to several event image frames, which may be conservative or non-conservative, and which may be captured for the entire duration of the event or a portion of its duration. If no event occurred, normal processing continues. However, if an event is detected in step 404, a warning image frame is generated in step 406.Warning image frames, as described here, correspond to event image frames but have a few different features.

[0040] In one embodiment, the warning image frame is a P-frame having predetermined leftward and rightward motion relative to at least a portion of the event image frame. Typically, this motion is implemented using motion vectors, as is well known to those skilled in the art. Both the amount and direction of motion can be configurable or predetermined, depending on the specific implementation. For example, encoder 304 can support both horizontal and vertical motion, also allowing any kind of oblique motion. As mentioned above, the hardcoded warning image frame is generated in software, for example, by IPP 302. However, in the case of a decoder decoding the hardcoded warning image frame, it should "look" as if it were encoded by encoder 304. Therefore, in the case of a decoder, there is no difference between the image frame encoded by encoder 304 and the hardcoded warning image frame generated in software, for example, by IPP 302. Thus, similarly, the hardcoded warning image frame can include a motion delta relative to the event image, the motion delta providing both horizontal and vertical motion. In a scenario where several event image frames exist (which is common), the corresponding warning image frames can be generated to include alternating motion. This will be explained in more detail below, but essentially, having alternating warning image frames have an image stream that includes alternating motion, which, when displayed on display 120, can achieve a “vibration” effect, which can help attract the operator's attention. It should be noted that the motion vector can be a global motion vector such that the entire warning image frame moves by a constant amount and in a constant direction relative to the event image frame. Alternatively, the motion vector can be applied only to a part of the event image frame, for example, a particular object of interest, thereby causing only that part of the warning image frame to move relative to the event image frame.

[0041] In another embodiment, the warning image frame is a P-frame having a predetermined color change relative to at least a portion of the event image frame. For example, the warning image frame may have a somewhat more colorful (e.g., somewhat reddish) representation relative to the event image frame. Of course, the colors do not have to be the same in different warning image frames. For example, one warning image frame may have a more reddish hue, while subsequent warning image frames may have a more blued hue. However, considering how people are accustomed to perceiving red as “danger” or “alert,” it can generally be a good choice to have reds of varying intensities within the warning image frames. Similar to motion vectors, the color change can be applied to the entire warning image frame or to a portion of the warning image frame. Similarly, the portion of the image frame does not necessarily have to be related to the content of the event image frame; for example, it could be the border around the warning image frame being highlighted. Such an example is shown in Figure 5, where the event occurs in section 204c of display 120.

[0042] In step 408, a warning image frame, i.e., a hardcoded warning image frame, is inserted after—typically immediately after—the event image frame in the encoded stream of image frames once generated, and the corresponding event image frame is marked as a hidden image frame. By essentially replacing the event image frame with the warning image frame, the original frame rate can still be maintained, and the content in the image stream will still look normal to the operator, apart from vibrations and / or color changes. Furthermore, once the operator acknowledges the event, the warning image frame can be removed, and the hidden features of the event image frame can also be removed, allowing the operator to determine whether the event that triggered the warning requires attention.

[0043] It should be noted that since events typically have a certain duration, there can be many event image frames associated with an event. In some embodiments, a warning image frame is generated for each event image frame, while in other embodiments, a warning image frame is generated only for a portion of the event image frame. A person skilled in the art can determine, based on the situation at hand, exactly how many event image frames should be used to generate a warning image frame. However, as a general guideline, regardless of how many warning image frames are generated, the warning image frames should include alternating directional motion and / or varying degrees of color intensity changes, because this makes the changes more noticeable to the operator. Furthermore, the way in which event image frames are selected allows for the configuration of various vibration patterns. For example, there may be three "gentle short-duration shakes," followed by a period of silence and display of the event image frames, and then a "large long-duration shake." In many ways, this is similar to what can be done with a mobile phone in vibration mode.

[0044] In some embodiments, the “oscillation” can continue after the event has ended and until the operator acknowledges the event. For example, in situations where the event is very short and the oscillation only occurred during the event, the operator may still miss the event. Therefore, to increase the likelihood of the operator detecting the event, the system can continue generating warning image frames until the operator acknowledges the event. This can be implemented, for example, by setting several “normal” image frames following the event frame to be “hidden” as well, and then appending a corresponding number of generated warning frames to the stream.

[0045] Step 410 determines whether there are any more image frames to process once a stream of image frames containing warning image frames has been generated. If so, normal image processing resumes in step 402, as described above. If there are no more image frames to process, for example, if the camera is turned off or goes to sleep, method 400 terminates.

[0046] To further illustrate the versatility of the present invention as described herein, consider the following examples. The event is triggered. The system is configured to alert the operator with a flashing red warning, which appears as a mixture of 0% to 100% red in the original image, repeating as follows: for example, 0% for event image frame #1, 50% for event image frame #2, 100% for event image frame #3, 50% for event image frame #4, 0% for event image frame #5, 50% for event image frame #6, and so on. • For some event image frames, particularly event image frames #2, #3, #4, #6, etc., a warning image frame is generated and appended to the stream of image frames, and event image frames #2, #3, #4, #6, etc. are marked as "hidden". However, for event image frames #1, #5, etc. (i.e., the warning image frame with 0% mixing is identical to the corresponding event image frame), a warning image frame does not need to be generated (and similarly, there would be no need to mark these image frames as "hidden" image frames).

[0047] In the above example, event image frames #1 and #5 will be encoded normally, while event image frames #2, #3, #4, and #6 will be encoded as hidden by the encoder. The first hardcoded warning image frame A1 will be generated by software, for example, IPP302, with 50% red mixed with respect to event image frame #2, and will be inserted by encoder 304 into the stream of encoded image frames in display order after event image frame #2. Similarly, the second hardcoded warning image frame A2 will be generated by IPP302 with 100% red mixed with respect to event image frame #3, and will be inserted by encoder 304 into the stream of encoded image frames in display order after event image frame #3. Furthermore, a third hardcoded warning image frame A3 will be generated by software, for example by IPP302, with 50% red mixed with respect to event image frame #4, and will be inserted by encoder304 into the stream of encoded image frames in display order after event image frame #4. Correspondingly, a fourth hardcoded warning image frame A4 will be generated by software, for example by IPP302, with 50% red mixed with respect to event image frame #6, and will be inserted by encoder304 into the stream of encoded image frames in display order after event image frame #6.Therefore, the stream of image frames will include the following image frames in display order: event image frame #1, event image frame #2 (hidden), first hardcoded warning image frame A1 (50% red compared to #2), event image frame #3 (hidden), second hardcoded warning image frame A2 (100% red compared to #3), event image frame #4 (hidden), third hardcoded warning image frame A3 (50% red compared to #4), event image frame #5, event image frame #6 (hidden), and fourth hardcoded warning image frame A4 (50% red compared to #6). As a person skilled in the art will understand, inserting an image frame into a stream of image frames may require updating one or more display order indicators to ensure the frame number of the image frame, possible references to one or more reference frames, correct decoding and correct decoding order of the image frames, and to ensure the correct display order when displaying the stream of image frames.

[0048] As this example shows, it is not necessary to generate a warning image frame for every event image frame in the stream of image frames; it may be sufficient to generate a warning image frame for only a subset of event image frames.

[0049] In one embodiment, there may be two separate “tracks,” one containing the original stream of image frames and another containing only the warning image frames. If the first track is set to “hidden,” only the stream of warning image frames will be displayed to the operator. As a result, the operator’s attention will be attracted by the “shaking still image” shown in the second track, after which the operator can switch to the first track, remove the hidden features, and observe what was actually captured by the camera.

[0050] Concluding comments While the above example focuses on the use of P-frames, it should be noted that the same general principles of the present invention are applicable to B-frames, which can reference both forward and backward in the GOP. However, since B-frames typically have higher memory requirements than P-frames, it will be preferable to use P-frames in most cases.

[0051] Furthermore, although the above examples have been described as individual embodiments in which motion or color change occurs within the warning image frame, there is nothing that would prevent these embodiments from being combined. For example, vibration and flashing effects can be generated, which can further increase the likelihood that the operator will quickly notice video material containing an event. In some embodiments, other effects can be applied. For example, image twisting or "warping" can be done as an alternative to (or in combination with) vibration and flashing. In addition, in some implementations, the type of warning can be changed if the warning is not perceived by the operator for a certain period of time. For example, if a "vibration" warning is not perceived, the warning can be changed to a flashing warning, etc.

[0052] It should also be noted that in some implementations, there may be various types of audible warnings (such as warning sounds or alarms) related to visual warnings displayed on the screen.

[0053] The systems (e.g., image processing pipelines and / or encoders) and methods disclosed herein can be implemented as software, firmware, hardware, or a combination thereof. In hardware implementations, the division of tasks between functional units or components as referred to above does not necessarily correspond to a division into physical units; conversely, one physical component can perform multiple functions, and a single task can be performed in coordination by several physical components.

[0054] Certain or all components may be implemented as software executed by a digital signal processor or microprocessor, or as hardware or as application-specific integrated circuits. Such software may be delivered on computer-readable media, which may include computer storage media (i.e., non-temporary media) and communication media (i.e., temporary media). As is well known to those skilled in the art, the term computer storage media includes together volatile and non-volatile removable and non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store desired information and that can be accessed by a computer.

[0055] The flowcharts and block diagrams in the figures illustrate the architecture, functions, and operations of possible implementations of the system, method, and computer program product according to various embodiments of the present invention. In this regard, each block in the flowcharts and block diagrams may represent a module, segment, or a predetermined portion of an instruction containing one or more executable instructions for implementing a specified logical function(s). In some alternative implementations, the functions described in the blocks may occur in a different order than that shown in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowchart examples, and combinations of blocks in the block diagrams and / or flowchart examples, may be implemented by a dedicated hardware-based system that performs a specified function or operation or a combination of dedicated hardware and computer instructions.

[0056] Those skilled in the art will recognize that the embodiments described above can be modified in many ways and the advantages of the present invention shown in those embodiments can still be used. Therefore, the present invention should not be limited to the embodiments shown, but rather should be defined only by the appended claims. Furthermore, the embodiments shown can be combined, as those skilled in the art will understand.

Claims

1. A method in a camera system (108) for processing a stream of image frames, The process includes generating a hardcoded warning image frame in response to detecting an event, wherein the hardcoded warning image frame is an inter-image frame and includes a motion delta and / or color change with respect to the event image frame, wherein the motion delta includes horizontal and / or vertical movement with respect to the event image frame, and the hardcoded warning image frame is generated by software to produce a desired change in the video stream when displayed to the operator. The method further comprises generating a stream of encoded image frames, wherein the hardcoded warning image frame is inserted into the stream of encoded image frames in display order after the encoded event image frame.

2. The method according to claim 1, further comprising encoding the event image frame as a hidden image frame.

3. The method according to claim 1 or 2, wherein the event is an event detected within an event image frame in a stream of image frames or an external event.

4. The method according to any one of claims 1 to 3, wherein the motion delta has a configurable size or a predetermined size.

5. The method according to any one of claims 1 to 4, wherein the motion delta is applied only to a portion of the warning image frame.

6. The method according to any one of claims 1 to 5, wherein the color change includes one or more of the following: a more colorful representation compared to the event image frame, a less colorful representation compared to the event image frame, and a representation in which the colors change compared to the event image frame.

7. The method according to any one of claims 1 to 6, wherein the color change is applied only to a portion of the warning image frame.

8. The method according to any one of claims 2 to 7, further comprising removing the hidden state of the event image frame and changing the state of the warning image frame to hidden in order to enable the operator to observe the event captured by the camera system in response to input from the operator.

9. The method according to any one of claims 1 to 8, wherein the warning image frame is one of a forward prediction frame (P frame) including a motion delta with respect to the event image frame, and a bidirectional frame (B frame) including a motion delta with respect to the event image frame.

10. The generation of the hardcoded warning image frame is The method according to any one of claims 1 to 9, comprising generating the hardcoded warning image frame based on the warning image frame generated outside the camera system (108).

11. The method according to any one of claims 1 to 10, wherein the event image frame is a reference frame for the picture group GOP.

12. A camera system (108), A lens (110) and an image sensor (112) configured to capture a stream of image frames, The system includes an image processing pipeline (302) configured to generate hardcoded warning image frames in response to detecting an event, The hardcoded warning image frame is an inter-image frame and includes a motion delta and / or color change with respect to the event image frame, the motion delta includes horizontal and / or vertical motion with respect to the event image frame, and the hardcoded warning image frame is generated by software to produce a desired change in the video stream when displayed to the operator. The camera system (108) further comprises an encoder (304) configured to generate a stream of encoded image frames, The hardcoded warning image frame is inserted into the stream of encoded image frames in the order of display after the encoded event image frame, camera system (108)

13. A computer program product for processing a stream of image frames captured by a camera, comprising a computer-readable storage medium implementing program instructions, wherein the computer-readable storage medium is not a transient signal itself, and the program instructions are executable by a processor to carry out the method. The method includes generating a hardcoded warning image frame in response to detecting an event, wherein the hardcoded warning image frame is an inter-image frame and includes a motion delta and / or color change with respect to the event image frame, wherein the motion delta includes horizontal and / or vertical motion with respect to the event image frame, and the hardcoded warning image frame is generated by software to produce a desired change in the video stream when displayed to an operator. The method further comprises generating a stream of encoded image frames, wherein the hardcoded warning image frame is inserted into the stream of encoded image frames in display order after the encoded event image frame, in a computer program product.

Images