Video monitoring with robust privacy masking

Abstract

To provide a robust privacy masking method.SOLUTION: A controller 130 interacts with an image processing chain 110 that processes a video stream 140 captured by a video camera 120. The image processing chain comprises: an object detection algorithm 114 which outputs a detection score for each frame to each image region; and a masking function 116 which is applied with a privacy mask according to the detection score. A controller maintains or expands the privacy mask 141 applied in the time of detected abrupt change over a period in which a scene is shot when the abrupt change in luminance of the scene shot by the video camera is detected. The maintenance or expansion of the privacy mask may be achieved by locally reducing a masking threshold applied by the masking function. The expansion of the privacy mask may be gradual over the time and related to an estimated velocity of a moving object in the scene.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and more particularly, to a robust privacy masking method under changing lighting conditions.

Background Art

[0002] Privacy masking refers to a technique for removing areas of an image or video frame where there is no legitimate monitoring target, such as in the case of a human face, keypad, vehicle number plate, etc. These areas can be removed by replacing (i.e., hiding) them with other data so that any personal data contained therein becomes unreadable, or can be corrected by image processing. One state-of-the-art privacy masking product relies on a underlying neural network. Video frames are analyzed one by one by the neural network. The neural network takes a video frame as input and outputs a detection score (e.g., a confidence level) indicating the probability that an object (such as a face, keypad, number plate, etc.) that requires masking is visible at different pixels / regions of the input video frame. To mask the video frame to protect privacy, a privacy masking threshold is applied to the detection score map, whereby pixels / regions of the video frame having a detection score above the privacy masking threshold are masked. The remaining pixels / regions of the video frame, where the detection score is below the privacy masking threshold, remain visible. State-of-the-art products use a fixed privacy masking threshold.

[0003] It has been noted that the underlying neural network can degrade in performance and become unstable, leading to reduced convergence, when lighting conditions change rapidly (e.g., artificial light is turned on and off). In particular, a temporary overall decrease in the neural network's detection score has been observed. For this reason, if the detection score decreases and the privacy masking threshold does not change, previously masked objects may become visible across multiple video frames until the neural network outputs a detection score of the expected magnitude again.

[0004] Unmasking masked objects in video, even briefly, is unacceptable from a privacy standpoint. For example, a few unmasked frames showing a person's face may be enough to identify that person and track them for the rest of the video based on their clothing, physical characteristics, etc. However, in monitoring applications, it is also unacceptable to use excessive precautions to deal with sudden changes in lighting, such as completely blacking out the video image (video signal). An intruder who notices that a monitoring system reacts in this way could use it to hide themselves. For example, an intruder could induce blacking out by flashing a light source towards the monitoring camera when entering the scene being monitored. Therefore, a balanced solution to the unmasking problem that satisfies both privacy concerns and monitoring concerns would be highly desirable. [Overview of the project]

[0005] One object of this disclosure is to make available a method and device that includes a masking function and interacts with an image processing chain configured to process a video stream captured by a video camera. A further object is to propose such a method and device that operates robustly under fluctuating lighting conditions. Yet another object is to propose such a method and device that has the ability to adequately mask moving objects imaged through rapid luminance fluctuations. A further object is to propose a method for realizing these techniques with limited interference to and reconstruction of the image processing chain. Yet another object is to formulate suitable criteria for use in such a method and device to automatically determine the beginning and end of a period with rapid luminance fluctuations. A particular object is to provide such a method and device for use in video monitoring applications.

[0006] At least some of these objectives are achieved by the present invention as defined by the independent claims. The dependent claims relate to advantageous embodiments.

[0007] A first aspect of this disclosure provides a method for interacting with an image processing chain. The image processing chain is configured to process a video stream captured by a video camera. The image processing chain includes an object detection algorithm that outputs a frame-by-frame detection score for each image region, and a masking function that applies a privacy mask according to the detection score. The image region may be a pixel or a group of pixels. The method includes detecting abrupt changes in the brightness of a scene. When such abrupt change is detected, the privacy mask applied at the time of the detected abrupt change is maintained or extended over a period of time.

[0008] When used in this disclosure, privacy masking operations may include removing or replacing image data within the image region to be masked. "Privacy mask" refers to the spatial extent of privacy masking in a given video frame, e.g., a set of image regions. "Brightness" does not necessarily have to be used in an objective physical sense (luminous intensity per unit area) and may further reflect the optical characteristics of the video camera, including the sensitivity of an internal image sensor, which may, under certain conditions, nonlinearly amplify the brightness of incident light. For example, abrupt changes in brightness may be detected based on abrupt brightening of the video image or based on different indirect criteria. Furthermore, brightness (or luminance) may also refer to a component of the image signal.

[0009] When the method having the above characteristics is performed, the privacy mask, determined and updated based on reliable detection score data (and therefore assumed to be adequately fitted to the captured scene up to abrupt changes in brightness), is maintained over a period corresponding to multiple further video frames. Alternatively, the privacy mask is extended, i.e., the privacy masking operation is applied to additional pixels or image areas. The inventors have understood that the position occupied by the privacy-masked image feature at the time of the detected abrupt change is most likely to be the future position of the same image feature, and therefore it is reasonable to maintain the privacy mask at least within the corresponding image area. It is recognized that the method steps outlined above are performed while the video camera is capturing the same scene. If the scene is significantly replaced or changed as a result of, for example, camera pan, tilt, or zoom (unless these actions are known and the resulting image changes are regularly offset), the maintenance or extension of the privacy mask may not achieve its intended effect. It is noteworthy that the described method, with appropriate adjustments, can protect the desired level of privacy throughout an entire episode of rapid brightness fluctuations without the need to take any drastic precautions such as complete blacking of the video image.

[0010] In some embodiments of the present invention, rapid changes in scene brightness can be indirectly detected by monitoring the average brightness, brightness histogram, brightness variance, exposure mismatch ΔE, or control variables related to the exposure of a video camera. It is particularly advantageous to monitor the rate of change of these quantities, such as those estimated over several past video frames.

[0011] Some embodiments of the present invention are conceived with particular attention to moving objects requiring masking, such as a person walking or a moving vehicle with a visible license plate. According to these embodiments, when a moving object to which a privacy mask is applied is detected, the masking function is made to expand the privacy mask around the moving object. More precisely, the privacy mask is expanded around the most recently known location of the moving object, as the reliability of the object detection algorithm may be temporarily reduced as a result of abrupt changes in brightness. Alternatively, the effect of equally expanding around the periphery may be achieved without knowing the exact location of the moving object at any given time, i.e., by determining the image regions to which the privacy mask is applied due to the moving object and expanding those image regions. Whichever of these options is implemented, the expansion of the privacy mask may be gradual over time. This takes into account the fact that the location of the moving object becomes gradually more uncertain over time and that it takes more time to move away from this initial location. The amount of expansion may be proportional to the estimated velocity of the moving object, which is estimated at the time of the detected abrupt change. The extension of the privacy mask may be limited to the estimated direction of motion of the moving object, which is estimated at the time of detected abrupt changes. The privacy mask, or the portion of the privacy mask associated with the moving object, may be translated in the estimated direction of motion.

[0012] Some embodiments of the present invention enable the integration of the invention into existing technologies in a particularly simple manner. These embodiments are feasible when the masking function in the image processing chain applies a masking threshold that can be configured independently for each image region. According to these embodiments, the masking function is then made to maintain or expand the privacy mask by reducing the masking threshold in any image region to which the privacy mask is applied when abrupt changes are detected. This realizes the invention with little to no need for costly and time-consuming modifications to the image processing chain.

[0013] A further group of embodiments focuses on the length of time for which a privacy mask is maintained or extended. The duration may be described as a recovery period for the video camera or an associated auto exposure loop. In one embodiment, the duration is predetermined (and may be set according to a specific heuristic described later). Alternatively, the duration is determined based on the rate of change of a control variable related to the video camera's exposure, such as average luminance, luminance histogram, luminance variance, exposure mismatch, or the rate of change, which may be estimated over consecutive frames of the video stream. Alternatively, the duration is determined based on the magnitude of the exposure mismatch when a sudden change in luminance is detected. Even more alternatively, the duration is determined based on how much the average luminance deviates from the setpoint average luminance when a sudden change is detected.

[0014] Another further embodiment provides a stop criterion for a period. More precisely, a period may be interrupted based on the determination that the exposure mismatch (or its absolute value) has returned to below a threshold. Alternatively or in addition, a period may be interrupted when the detection score in an image region rises to a level at which the masking function would apply a privacy mask in the same image region at the time of the abrupt change in which the object detection algorithm was detected, and this behavior can be perceived as evidence that the video camera or auto exposure loop has recovered, so that the object detection algorithm is outputting a detection score of an expected magnitude. At least some of these start and stop criteria can be robustly automated, and they can also be easily fine-tuned by adjusting thresholds, etc.

[0015] In a second embodiment, a controller is provided configured to interact with the image processing chain of the type outlined above. In this sense, the controller has processing circuitry and memory. The controller is configured to detect abrupt changes in the brightness of the scene being captured by the video camera over the period that the video camera is capturing the scene, and to maintain or expand the privacy mask applied at the time of the detected abrupt change. This second embodiment of the present invention generally shares the effects and advantages of the first embodiment and can be realized with corresponding technical modifications.

[0016] The present invention further relates to a computer program that includes instructions causing a computer, in particular a controller, to carry out the methods described above. The computer program may be stored in or distributed on a data carrier. As used herein, “data carrier” may be a transient data carrier, such as a modulated electromagnetic wave or light wave, or a non-transient data carrier. Non-transient data carriers include volatile and non-volatile memories, such as magnetic, optical, or solid-state permanent and non-permanent storage media. Again within the scope of “data carrier,” such memories may be permanently mounted or portable.

[0017] In general, all terms used in the claims should be interpreted according to their ordinary meanings in the art unless otherwise expressly defined herein. All references to “an element, apparatus, component, means, step, etc.” should be broadly interpreted as referring to at least one example of such element, apparatus, component, means, step, etc. unless otherwise expressly defined herein. The steps of any method disclosed herein do not necessarily have to be performed in the exact order described unless expressly defined otherwise.

[0018] Hereinafter, aspects and embodiments of the present invention will be described with reference to the accompanying drawings as examples. [Brief explanation of the drawing]

[0019] [Figure 1] This figure shows a video camera having an integrated image processing chain and a controller configured to interact with the image processing chain in a way that ensures privacy masking through episodes of rapid brightness fluctuations. [Figure 2] This is a flowchart of the method according to the embodiments described herein. [Figure 3] This figure shows a video camera having an external image processing chain and a controller configured to interact with the image processing chain in a way that ensures privacy masking through episodes of rapid brightness fluctuations. [Figure 4] This figure shows three privacy-masked video frames, acquired immediately before a sudden change in brightness (t=0) and at two subsequent time points (t=1, t=2), illustrating how gradually expanding the privacy mask can protect the masking of moving objects. [Figure 5] This is a diagram showing the time evolution of the exposure mismatch indicator. [Modes for carrying out the invention]

[0020] Hereinafter, aspects of the present disclosure will be described more fully with reference to the accompanying drawings that show specific embodiments of the invention. However, these aspects may be embodied in many different forms and should not be construed as limited. Rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of all aspects of the invention to those skilled in the art. Throughout the specification, like numbers refer to like elements.

[0021] FIG. 1 shows, at a high level, a video camera 120 configured to output a video stream 140. The video camera 120 may be a digital video camera 120 adapted for monitoring or surveillance applications. The video camera 120 may be fixedly mounted. In the video camera 120, an image sensor 121 provides a raw video stream to an image processing chain 110. The image processing chain 110 may include an object detection algorithm 114 configured to output a per-frame detection score for each image region. The image processing chain 110 may further include a masking function 116 that applies a privacy mask 141 according to the detection score.

[0022] The object detection algorithm 114 may be configured or trained to detect objects that require masking, such as human faces, keypads, or vehicle license plates. The detection score D(i) of the object detection algorithm 114 for an image region i may be a confidence level or probability that can be interpreted as the likelihood that an object of the detected type exists in the image region. Alternatively, the detection score may point to a scale [0,1] in arbitrary units, the endpoints of which may conceptually convey levels such as "no indication that an object exists" (0) and "highest certainty that an object exists" (1). The detection scores are provided as a table or map of values ​​that associate each image region with a detection score value. Object detection algorithms that provide such detection scores are well known and commercially available. The masking function 116 may perform a thresholding operation such that a privacy mask is applied to image regions where the detection score is greater than or equal to the masking threshold D0. A simplified example with only nine image regions per frame and a constant masking threshold D0 = 0.70 is shown in Table 1. The masking applied to image regions 5, 7, and 8 in TIFF0007839714000001.tif73170 may include removing visual features from the video image. For example, the areas corresponding to image regions 5, 7, and 8 may be cropped or trimmed, and the image data may be permanently removed before the video stream exits the video camera 120. Alternatively, the same image regions may be overlaid with a static masking pattern and the original image data replaced with the masking pattern. Yet another alternative is that the image data in the image regions is processed to be blurred, pixelated, or otherwise unrecognizable. Pixelation may include dividing the area into smaller blocks and replacing the image data of each block with a single value, such as the average of the pixels in the block or one of the pixel values in the block. Yet another option is that the masking function 116 applies the privacy mask without modifying the video stream itself, but instead adds (required) masking instructions to the video stream that will be executed by a receiver-side playback application. Thus, the raw video stream provided by the image sensor 121 may contain information other than, or different from, the processed video stream 140 output from the video camera 120.

[0023] The image processing chain 110 may optionally include an automatic exposure (AE) algorithm 112 configured to reduce exposure mismatch by gradually increasing and decreasing control variables related to the exposure of the video camera. Video frames captured by the video camera 120 have exposure mismatch if they do not correspond to the desired (or target or setpoint) exposure level. Exposure mismatch may be represented as an indicator ΔE, whose magnitude reflects the severity of the mismatch and whose sign corresponds to overexposure or underexposure. The control variables related to exposure may be exposure time, image sensor gain, or similar variables, if the second option is used. A common range for exposure time is 1 to 30 ms. The AE algorithm 110 may include a closed control loop, for example, acting as a proportional (P) controller. The P controller may be stateful (adaptive) or stateless. The control loop may further include integral (I) terms or derivative (D) terms, or both. The I and D terms may refer to a history (e.g., a sliding window) of the exposure mismatch indicator ΔE. In addition to adjusting exposure-related control variables, the AE algorithm 112 may apply compensation processing to the raw video stream to brighten underexposed frames and darken overexposed frames. Depending on the characteristics of the object detection algorithm 114, the video stream can be supplied to the object detection algorithm 114 before the compensation processing by the AE algorithm 112, or after the compensation processing as shown in Figure 1.

[0024] Figure 1 further shows a controller 130 configured to interact with the image processing chain in such a way as to ensure privacy masking through episodes of rapid brightness fluctuations. The primary functions of the controller 130 are to analyze the video stream to detect abrupt changes in brightness, and, in the case of a positive detection, to maintain or extend the privacy mask applied at the time of the detected abrupt change over a period of time.

[0025] In the example shown in Figure 1, the controller 130 performs detection on the raw video stream provided by the image sensor 121. This enables detection that is highly responsive to luminance fluctuations. Another possible option is to perform detection on the video stream immediately downstream of the AE algorithm 112, i.e., after compensation processing has been performed to normalize the brightness when a video image is present. The exposure-compensated video stream corresponds to the input data for the object detection algorithm 114. Therefore, detection of luminance changes based on an exposure-compensated video stream can avoid certain false detection positives if the AE algorithm 112 succeeds in compensating for some of the abrupt luminance fluctuations occurring in the scene. In other words, there is no need to dynamically manage small or otherwise harmless luminance fluctuations that do not destabilize the object detection algorithm 114.

[0026] Figure 1 shows two alternative options that the controller 130 may use to achieve the maintenance or enhancement of the privacy mask. The controller 130 may reconfigure the masking function 116 (upper horizontal arrow), the controller 130 may process the video stream (lower horizontal arrow) before it is output from the video camera 120, or a combination of these. In the second option, if the controller 130 processes the video stream to maintain or enhance the privacy mask, the controller 130 does not need to be authenticated to reconfigure the masking function 116 or act on it in another way, but the controller 130 may have read access to the output data or input data or the configuration parameters of the masking function 116 so that the controller 130 can determine or reconstruct the privacy mask applied in the event of a sudden change detected.

[0027] To satisfy the above-mentioned functionality, the controller 130 may include input and output interfaces (not shown), a processing circuit 131, and a memory 132. If the processing chain 110 includes an AE algorithm 112, as shown in Figure 1, the controller 130 may be configured to receive data from the AE algorithm 112, such as exposure mismatch ΔE or control variables related to exposure. Figure 1, like Figure 3, shows the controller 130 and other components in block diagram form, and it should be noted that the blocks primarily reflect the functions or activities to be performed. The physical resources on which these functions or activities are realized do not need to be organized or structured as the blocks in the diagrams suggest. In fact, all or most of the blocks seen in Figures 1 and 3 may be realized as software code that runs on general processing resources.

[0028] In the example shown in Figure 1, the image processing chain 110 is integrated into the video camera 120, so that the video stream does not leave the video camera 120 until privacy masking is applied, which protects against unauthorized persons attempting to gain access to video streams that meet those conditions upstream of the masking, such as the raw video stream.

[0029] A possible alternative architecture is shown in Figure 3, where the image processing chain 110 and controller 130 are located outside the video camera 120. As suggested by their dashed outlines, the image processing chain 110 and controller 130 do not need to correspond to physical units and can be implemented in distributed and / or networked (cloud) processing resources that have access to the video stream and / or are certified to reconfigure or inspect the masking function 116. In particular, the image processing chain 110 and controller 130 can be run by an external server such as a video management system. The architecture in Figure 3 may optionally include a data connection from the AE algorithm 112 of the processing chain 110 to the controller 130, as shown in Figure 1.

[0030] Although not explicitly shown in Figures 1 and 3, a hybrid structure is also possible in which at least some components of the image processing chain 110 are internal components of the video camera 120 and the controller 130 is external. For example, the AE algorithm 112 may be an internal component of the video camera 120. The video camera 120 may then incorporate the exposure mismatch ΔE and / or exposure-related control variables as metadata in the video stream or transmit them on a separate frame-tagged (frame-stamped) channel so that the controller 130 can access them. This is possible in the Exif image file format, which allows the exposure settings for a frame to be read.

[0031] A further alternative architecture is one in which the masking function 116 applies a privacy mask without modifying the video stream itself, but provides the video stream with masking instructions to be executed by the video playback application. In this setup, the controller 130's actions may be to modify or replace these masking instructions so that the privacy mask applied in the event of a sudden change is maintained or extended over a period of time.

[0032] Referring to Figure 2, a method 200 for interacting with the image processing chain 110 is described next. Method 200 may be implemented with a controller 130 of the type illustrated in Figures 1 and 3.

[0033] In detection step 212, which should be performed while the video camera 120 is capturing the scene, the controller 130 attempts to detect abrupt changes in the scene's brightness. In a special case, this includes abrupt changes in brightness that affect only a portion of the scene. The controller 130's detection may include monitoring one or more quantities for consecutive frames of the video stream, such as average brightness, brightness histogram, brightness variance (e.g., variance across each video frame), exposure mismatch ΔE, and control variables related to the exposure of the video camera 120. These quantities may refer to the raw or processed video stream, as read at a point further downstream of the image sensor 121. Monitoring may be performed separately for different sub-areas of the video frame. Abrupt changes in brightness may be considered present if the rate of change of the monitored quantity exceeds a threshold, such that the monitored quantity as a whole has changed more than a threshold increment from the previous frame. Alternatively, abrupt changes may be detected based on the criterion that the monitored quantity has changed more than a threshold increment over n0 recent frames (n0≧2).

[0034] Such monitoring of exposure mismatch ΔE (a scalar value) is shown in Figure 5. The beginning of a period of length T, during which the privacy mask should be maintained or expanded, is marked by a rapid increase in exposure mismatch ΔE at t=0.

[0035] The rate of change of the luminance histogram is the vector norm (L p ,l p The rate of change in the luminance histogram may be observed using distance criteria, such as the luminance histogram or a probability norm (Bhattacharya distance, Kullback-Leibra distance, and many further options). Alternatively, the rate of change in the luminance histogram may be observed by tracking the change in the histogram mean or histogram variance (these statistics refer to the frequency of each bin in the histogram), or by tracking the movement of a selected reference point on the histogram.

[0036] The control variables related to the exposure of the video camera 120 may be exposure time, image sensor gain, or similar variables. As described above, the control variables related to exposure may be adjusted in a closed loop by the AE algorithm 112, whose task is to monitor exposure mismatches and adjust the control variables related to exposure as appropriate. As a result, ignoring transients, higher brightness in the scene will eventually lead to shorter exposure times, and vice versa. Thus, the option of monitoring the control variables related to exposure provides a useful indirect method for detecting abrupt changes in brightness in the scene, thereby avoiding the duplication of the existing brightness monitoring in the AE algorithm 112. Advantageously, in the processing chain 110 of the AE algorithm 112, where the control loop for specifically reacting to abrupt changes includes a differential (D) term, the adjusted control variables related to exposure may reflect the underlying brightness fluctuations, including specific emphasis.

[0037] In the improved example of step 212, abrupt changes in brightness are detected separately for different blocks (or sections) of the image. More precisely, it can be concluded that the first block of the video image experiences such abrupt changes, while the second and third blocks do not. Therefore, the actions taken according to this method 200 in response to abrupt changes can be limited to the first block, as will be discussed further later. This is particularly relevant in high dynamic range (HDR) scenes where multiple exposures may be used.

[0038] In connection with the detection step 212, an optional period determination step 214 may be performed. In this step 214, the length T of the period for which the privacy mask is maintained or extended is determined based on the magnitude of the rate of change or deviation of one or more of the quantities monitored in the detection step 212. Specifically, the determination may depend on the following: - The rate of change of control variables related to video camera exposure, such as average luminance, luminance histogram, luminance variance, exposure mismatch ΔE, or the rate of change of consecutive frames in a video stream. - The magnitude of exposure mismatch ΔE during the detected rapid change, and / or - The magnitude of the deviation of the average brightness from the setpoint average brightness during detected rapid changes. The setpoint average brightness may correspond to a desired average brightness. It may also form part of the configuration parameters of the AE algorithm 112.

[0039] In embodiments where the maintenance / extension period is not determined during the execution of Method 200, a predetermined length T may be used. One example of a heuristic for setting the length of the predetermined period is to relate the length of the predetermined period to the dynamic properties of the AE algorithm 112, such as the duration of the impulse response. The duration of the impulse response relates to how quickly the AE algorithm 112 reacts to exposure mismatch of a given magnitude, and the duration of the impulse response may then be determined according to the configured values ​​of various control gains in the AE algorithm 112. The duration of the impulse response may refer to a period in which the absolute value of the impulse response is different from zero by at least a positive threshold ε>0, i.e., the long tail is ignored. With this in mind, a relatively long duration of the impulse response may indicate that the image processing chain 110 takes a relatively long time to recover from abrupt changes in scene brightness, so the privacy mask applied during detected abrupt changes needs to be maintained (or extended) for a relatively long time to allow the object detection algorithm 114 to stabilize. Conversely, for similar reasons, if the impulse response has a relatively short duration, the time for which a maintained privacy mask is required may be relatively short.

[0040] In the maintain / extend step 216, which is performed while the video camera 120 is still capturing substantially the same scene, one or more actions are taken to ensure that the privacy mask applied at the time of the detected abrupt change remains applied or extended over the aforementioned period.

[0041] As described above, such action may include processing the video stream. If the masking function 116 is bypassed, the processing may include adding a full privacy mask that covers at least the same image region as the detected abrupt change. If the alternative step 216 involves processing the video stream at a point downstream of the masking function 116, the processing may include evaluating whether the privacy mask applied by the masking function 116 still covers at least the same image region as the detected abrupt change. If it does not, privacy masking is added to the image region that does not have a privacy mask.

[0042] Alternatively, the action in step 216 may include modifying the settings of the masking function 116. Specifically, assuming that the masking function 116 uses a masking threshold D0 = D0(i), which can be configured independently for each image region i of the video frame F, the masking function 116 may maintain or expand the privacy mask by reducing the masking threshold in any image region to which the privacy mask is applied when a sudden change is detected. If a privacy mask is applied to an image region i ∈ I (I ⊆ F) when a change is detected, the reduction of the masking threshold may correspond to an assignment D0(i) ← α for all i ∈ I, where α is a small constant value such as -∞ or 0, or the minimum specified value of the detection score. The assignment is, The masking threshold D0(i) for TIFF0007839714000002.tif7170 remains unchanged. Thus, the threshold setting operation in the masking function 116 applies masking to all image regions i∈I, regardless of the detection score D(i) received from the object detection algorithm 114.

[0043] Refer to Table 1 to illustrate how the settings of the masking function 116 can be modified to achieve step 216. Table 1 is assumed to reflect the situation immediately before a sudden change in brightness occurs. As shown in Table 1, the privacy mask was applied to the image region of set I = {5, 7, 8}. The object detection algorithm 114 outputs a reduced detection score until it recovers from the sudden change. To ensure that the privacy mask is still applied, a small constant value α (e.g., 0.00 or 0.02) is assigned as the masking threshold for all image regions in I, as shown in Table 2. TIFF0007839714000003.tif68170

[0044] As outlined above, when the detection step 212 is implemented on a block-by-block basis, it can be advantageous to apply the masking / expansion step 216 only to the blocks where abrupt changes in brightness are detected. In particular, the masking threshold of the masking function 116 only needs to be reduced in the image regions that overlap with those blocks.

[0045] The masking / extension step 216 may be performed until the period has elapsed, i.e., over T time units. In some embodiments, method 200 further includes an event-based interruption step 218 of the period. More precisely, - Event 1: Exposure mismatch returns to below the threshold |ΔE|≦β - Event 2: The detection score D(i) in the image region has risen to a level where the masking function applies a privacy mask within the same image region where a rapid change was detected. and / or - Event 3: The rate of change of a control variable related to video camera exposure, such as average luminance, luminance histogram, luminance variance, exposure mismatch ΔE, or a succession of frames in the video stream, falls below a threshold. If any of the above is determined, step 216 is aborted. At this point, a new cycle 212 can be started to detect abrupt changes in brightness in the scene. With respect to event 1, the threshold value β≧0 can be set by trial and error. More precisely, the video camera 120 may be exposed to rapid brightness fluctuations if it is composed of different candidate values ​​β1, β2, ..., during which the operator monitors whether any undesirable unmasking occurs, such as faces, keypads, license plates, etc. The best candidate value that achieves this can be selected. Event 2 can be formulated as the following set of inequalities. ∀i∈I, D(i)≧D0(i) In the formula, I⊆F is the set of image regions to which a privacy mask was applied at the time of the detected abrupt change. The set of inequalities can be evaluated collectively (all must be satisfied for normal operation to resume) or one by one. In the second option, normal operation may resume in image region i as soon as it is found that the detection score has risen locally to a level that reaches a threshold. In a further modification of this embodiment, event 2 may be relaxed to a criterion such that D(i)≧D0(i) is satisfied for at least some percentage of the image region in I, for example, 80% or 90%. The percentage may correspond to a degree of temporary unmasking that is considered acceptable in a common use case.

[0046] Some embodiments of method 200 are particularly adapted to handle moving objects in a scene. More precisely, if it is detected (210) that the scene contains at least one moving object to which a privacy mask is applied during a rapid change in brightness, the privacy mask is extended around the moving object. The detection of moving objects 210 can follow an indirect policy in which the center of the privacy mask (or the centers of disparate components of the privacy mask) is tracked over time. This eliminates the need to apply dedicated motion detection in addition to the object detection algorithm 114.

[0047] Since the object detection algorithm 114 cannot be assumed to provide a useful detection score immediately after a rapid change in brightness, the privacy mask may be extended around the last known position of the moving object in the video frame. This may be achieved in a position-agnostic manner as shown below. Again, it is assumed that the privacy mask is applied to the image region of the index set I. The open neighborhood of I is defined for any r > 0 as follows: I r ={j|For some i∈I, dist(j,i) <r} In the formula, dist(·,·) is a distance function representing the distance between two image regions. Recalling Tables 1 and 2, which point to a simplified conceptual example, the total number of image regions in a video frame is at least an order of magnitude less than what would be a suitable masking granularity for a commercially available monitoring video camera. For example, an image region may be an individual pixel in the frame, or a group of squares such as 4, 16, or 25 pixels, respectively. If I refers only to moving objects, then the work 216 extending the privacy mask is neighbor I r This may correspond to applying masking to all image regions. This can be achieved, for example, by reducing the masking threshold around moving objects. Specifically, a small constant value α is such that all i∈I r It is assigned as the masking threshold D0(i) for . The radius r may be a specified constant r=r0. Alternatively, the expansion of the privacy mask is gradual with time. If the time at which a rapid change in brightness occurs is denoted as t=0, then method 200 is set I r(t) The region within the function may be masked, and r(t) is a non-decreasing function. For example, an affine function may be used. r(t) = at + b In the equation, a > 0 and b ≥ 0.

[0048] In a further development of this embodiment, Method 200 further includes estimating the velocity |ν| of a moving object in a rapid change 210.1 and making the velocity at which the privacy mask is extended linear or nonlinear in relation to the velocity estimation. More precisely, the privacy mask extends to neighbor I r(t) When applied to this, the growth coefficient a of the affine function r(t) may be related to |ν|. The relationship may be linear or affine. As ζ, η > 0, a = ζ|ν| + η Using affine relations such as those mentioned above, it is possible to accurately capture the dual contribution from both the uncertainty of the position that grows over time and the velocity of movement |ν|.

[0049] In a further development of this embodiment, velocity estimation 210.1 further includes estimating the direction of motion of a moving object during rapid changes, thereby making an estimate of the motion vector ν available. This estimate may be used for two purposes. On the one hand, the expansion of the privacy mask can be substantially restricted to the estimated direction of motion by, for example, adding an image region to the front end of the privacy mask rather than the rear end. On the other hand, the privacy mask can be translated in the estimated direction of motion, in other words, the privacy mask is expanded in the direction of motion and reduced in the opposite direction. Each option tends to track the moving object with reasonable accuracy during the period without obstructing the field of view by unnecessarily expanding the privacy mask.

[0050] Figure 4 shows the gradual expansion of the privacy mask 141 resulting from the detection of a walking person's face. The width of the privacy mask 141 (horizontal spread of the video frame) at a rapid change in brightness (t=0) is in w0 units. During the period in which step 216 is performed, the width of the privacy mask 141 is gradually increased at t=1 and t=2 according to the function of the following equation. w(t) = w0 + w1t In the formula, w1>0 is a constant. The privacy mask 141 remains centered at the same image point, which roughly corresponds to the position of the walking person's face during rapid changes. As shown in the figure, the gradual expansion of the privacy mask is as follows: r This can be achieved without using construction.

[0051] According to the first further development described above, the growth coefficient w1 is related to the estimated velocity |ν| of the walking person. Velocity refers to movement as seen in video images and may be expressed in pixels per second. According to the second further development described above, the direction of movement of the person is estimated and found to correspond approximately to the Cartesian vector (1,0). Under this second development (not shown in Figure 4), it is possible to extend the privacy mask 141 in the width direction without extending it in the height direction.

[0052] The aspects of this disclosure have been described above primarily with reference to several embodiments. However, as will be readily apparent to those skilled in the art, other embodiments not disclosed above are equally possible within the scope of the invention as defined by the appended claims. [Explanation of Symbols]

[0053] 110 Image Processing Chain 112 Automatic Exposure (AE) Algorithm 114 Object Detection Algorithms 116 Masking function 120 video cameras 121 Image Sensor 130 Controllers 131 Processing Circuit 132 memory 140 video streams 141 Privacy Mask 200 ways 210 Detection Step 210.1 Velocity Estimation Step 212 Detection Steps 214. Period Determination Step 216 Maintenance / Expansion Step 218 Interruption Step

Claims

1. A method (200) for interacting with an image processing chain (110) configured to process a video stream captured by a video camera (120), wherein the image processing chain (110) An object detection algorithm (114) that outputs a frame-by-frame detection score for each image region, A masking function (116) that applies a privacy mask (141) according to the detection score, and Includes, The masking function is configured to apply the privacy mask to image regions where the detection score exceeds a masking threshold configurable for each image region. While the aforementioned video camera is capturing the scene, Detecting a sudden change in brightness of the aforementioned scene (212), In response to detecting the abrupt change in brightness, the privacy mask is maintained or expanded (216), which includes reducing the masking threshold of the masking function in the image region where the privacy mask was applied at the time of the detected abrupt change, so that the masking function applies masking to each of the image regions regardless of the detection score received from the object detection algorithm (114). This is executed. method.

2. Detecting a sudden change (212) with respect to consecutive frames of the video stream, Average brightness, Brightness histogram, Brightness dispersion, exposure mismatch, Control variables related to the exposure of the video camera This includes monitoring one of the quantities of The method according to claim 1.

3. (210) further includes detecting whether the scene includes a moving object to which the privacy mask is applied, The privacy mask is extended around the moving object. The method according to claim 1.

4. The method according to claim 3, wherein the privacy mask is gradually expanded over time around the moving object.

5. The method further includes estimating the velocity of the moving object (210.1), The privacy mask expands around the moving object in proportion to the estimated velocity. The method according to claim 3.

6. The estimation of the velocity (210.1) includes estimating the direction of motion of the moving object, The extension of the privacy mask is substantially restricted to the estimated direction of motion, and / or the privacy mask is translated in the estimated direction of motion. The method according to claim 5.

7. The method according to claim 3, further comprising reducing the masking threshold around the moving object.

8. The method according to claim 1, wherein the length of the period to be maintained or extended is predetermined.

9. The image processing chain (110) further includes an automatic exposure algorithm (112) configured to reduce exposure mismatch by gradually increasing and decreasing the exposure time control variable of the video camera. The length of the aforementioned period is related to the dynamic nature of the automatic exposure algorithm. The method according to claim 8.

10. The rate of change of the average brightness, brightness histogram, brightness variance, exposure mismatch, or control variable related to the exposure of the video camera with respect to consecutive frames of the video stream, The magnitude of the exposure mismatch during the detected rapid change, and The magnitude of the deviation of the average brightness from the setpoint average brightness during the detected rapid change. The method according to claim 1, further comprising determining the length of the period to be maintained or extended based on (214).

11. The exposure mismatch has returned to below the threshold, and, The detection score in the image region has risen to a level where the masking function applies the privacy mask within the same image region as the detected rapid change. The method according to claim 1, further comprising interrupting the period to be maintained or extended in response to one or more decisions of (218).

12. The method according to claim 1, wherein the video camera (120) is fixedly mounted.

13. A non-temporary computer-readable storage medium, which, when executed on a device having processing capabilities, stores instructions for performing the method according to any one of claims 1 to 12.

14. A controller (130) configured to interact with an image processing chain (110) configured to process a video stream captured by a video camera (120), An object detector that analyzes multiple image regions and outputs a frame-by-frame detection score for each image region, An object masking device that applies a privacy mask (141) according to the detection score, and Includes, The object masking device is further configured to apply the privacy mask to image regions where the detection score exceeds a masking threshold configurable for each image region. The controller comprises a processing circuit (131) and a memory (132), The video camera detects a sudden change in brightness of the scene being captured, While the video camera is capturing the scene, in response to detecting a sudden change in brightness, the object masking device maintains or expands the privacy mask, which includes reducing the masking threshold in the image region where the privacy mask was applied at the time of the detected sudden change, so that the object masking device applies the mask to each of the image regions regardless of the detection score received from the object detector. It is configured in such a way. controller.

Images