Video monitoring with robust privacy masking

Abstract

To provide a robust privacy masking method.SOLUTION: A controller 130 interacts with an image processing chain 110 that processes a video stream 140 captured by a video camera 120. The image processing chain comprises: an object detection algorithm 114 which outputs a detection score for each frame to each image region; and a masking function 116 which is applied with a privacy mask according to the detection score. A controller maintains or expands the privacy mask 141 applied in the time of detected abrupt change over a period in which a scene is shot when the abrupt change in luminance of the scene shot by the video camera is detected. The maintenance or expansion of the privacy mask may be achieved by locally reducing a masking threshold applied by the masking function. The expansion of the privacy mask may be gradual over the time and related to an estimated velocity of a moving object in the scene.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to the field of image processing, and more particularly to a privacy masking method that is robust under changing lighting conditions. [Background technology]

[0002] Privacy masking refers to a technique for removing areas of an image or video frame where legitimate monitoring targets are not present, such as in the case of a human face, a keypad, a vehicle license plate, etc. These areas can be removed by replacing them with other data (i.e., obscuring them) or modified through image processing so that any personal data contained therein is unintelligible. One state-of-the-art privacy masking product relies on an underlying neural network. Video frames are analyzed one at a time by a neural network. The neural network takes the video frames as input and outputs detection scores (e.g., confidence levels) that indicate the probability that an object requiring masking (e.g., a face, a keypad, a license plate, etc.) is visible in different pixels / regions of the input video frame. To mask a video frame to protect privacy, a privacy masking threshold is applied to the detection score map, whereby pixels / regions of the video frame with detection scores above the privacy masking threshold are masked. The remaining pixels / regions of the video frame with detection scores below the privacy masking threshold remain visible. State-of-the-art products use a fixed privacy masking threshold.

[0003] It has been noted that the underlying neural network can perform poorly when lighting conditions change rapidly (e.g., artificial lights are turned on and off), resulting in instability and poor convergence. In particular, a temporary overall decrease in the neural network's detection score has been observed. If this occurs and the privacy masking threshold remains unchanged, previously masked objects may become visible across multiple video frames until the neural network again outputs a detection score of the expected magnitude.

[0004] Even brief unmasking of a masked object in a video is unacceptable from a privacy perspective. For example, a few unmasked frames in which a person's face is visible may be sufficient to identify the person and track them for the remainder of the video based on their clothing, physical characteristics, etc. However, in monitoring applications, taking excessive precautions to address sudden lighting fluctuations, such as completely blacking out the video image (video signal), is also unacceptable. An intruder who notices that the monitoring system reacts in this way could exploit it to conceal themselves. For example, an intruder could induce blacking by flashing a light source toward the monitoring camera when entering the monitored scene. Therefore, a balanced solution to the unmasking problem that satisfies monitoring concerns as well as privacy concerns would be highly desirable. Summary of the Invention

[0005] One objective of the present disclosure is to make available a method and device that includes a masking function and interacts with an image processing chain configured to process a video stream captured by a video camera. A further objective is to propose such a method and device that operates robustly in varying lighting conditions. Yet another objective is to propose such a method and device that has the ability to properly mask moving objects imaged through rapid brightness fluctuations. A further objective is to propose a method for implementing these techniques with limited interference with and reconfiguration of the image processing chain. Yet another objective is to formulate suitable criteria for use in such a method and device that automatically find the beginning and end of periods with rapid brightness fluctuations. A particular objective is to provide such a method and device for use in video monitoring applications.

[0006] At least some of these objects are achieved by the invention as defined by the independent claims. The dependent claims relate to advantageous embodiments.

[0007] In a first aspect of the present disclosure, a method for interacting with an image processing chain is provided. The image processing chain is configured to process a video stream captured by a video camera. The image processing chain includes an object detection algorithm that outputs a frame-by-frame detection score for each image region, and a masking function that applies a privacy mask in response to the detection score. The image region may be a pixel or a group of pixels. The method includes detecting an abrupt change in luminance of a scene. When such an abrupt change is detected, a privacy mask applied at the time of the detected abrupt change is maintained or expanded over a period of time.

[0008] As used in this disclosure, a privacy masking operation may include removing or replacing image data within a masked image region. A "privacy mask" refers to the spatial extent of privacy masking in a given video frame, e.g., a collection of image regions. "Brightness" may not necessarily be used in an objective physical sense (light intensity per unit area) but may also reflect the optical characteristics of the video camera, including the sensitivity of the internal image sensor, which may nonlinearly amplify the brightness of incident light under certain conditions. For example, a sudden brightness change may be detected based on a sudden increase in brightness in the video image or based on different indirect criteria. Furthermore, brightness (or brightness) may refer to a component of an image signal.

[0009] When a method having the above-described features is performed, the privacy mask determined and updated based on reliable detection score data (and therefore can be assumed to be adequately adapted to the captured scene until the abrupt change in luminance) is maintained for a period corresponding to multiple additional video frames. Alternatively, the privacy mask is expanded, i.e., the privacy masking operation is applied to additional pixels or image regions. The inventors have recognized that the location occupied by the privacy-masked image feature at the time of the detected abrupt change is the most likely future location of the same image feature, and therefore it makes sense to maintain the privacy mask at least within the corresponding image region. It is recognized that the method steps outlined above are performed while a video camera is capturing the same scene. If the scene is significantly displaced or changed, for example as a result of panning, tilting, or zooming the camera (unless these actions are known and the resulting image changes are normally compensated for), maintaining or expanding the privacy mask may not achieve its intended effect. It is noted that, with suitable adjustments, the described method can preserve a desired level of privacy throughout episodes of rapid brightness fluctuations without the need to take any drastic precautions such as completely blacking out the video image.

[0010] In some embodiments of the present invention, sudden changes in scene luminance can be detected indirectly by monitoring the average luminance, luminance histogram, luminance variance, exposure mismatch ΔE, or control variables related to the video camera exposure. It is particularly advantageous to monitor the rate of change of these quantities, as estimated over the past few video frames.

[0011] Some embodiments of the present invention are specifically designed to address moving objects requiring masking, such as a walking person or a moving vehicle with a visible license plate. According to these embodiments, when a moving object to which a privacy mask is applied is detected, the masking function is adapted to expand the privacy mask around the moving object. Specifically, since sudden changes in brightness can temporarily reduce the reliability of object detection algorithms, the privacy mask is expanded around the last known position of the moving object. Alternatively, the effect of uniform expansion around the periphery may be achieved without knowing the exact position of the moving object at any given time, i.e., by determining image regions to which a privacy mask is applied due to the moving object and expanding those image regions. Regardless of which of these options is implemented, the expansion of the privacy mask may be gradual over time. This takes into account the fact that the position of a moving object becomes increasingly uncertain over time and takes longer to move away from its initial position. The amount of expansion may be proportional to the estimated velocity of the moving object, which is estimated at the time of the detected sudden change. The expansion of the privacy mask may be restricted to the estimated direction of motion of the moving object, the direction being estimated at the time of the detected abrupt change, and the privacy mask, or a portion of the privacy mask associated with the moving object, may be translated in the estimated direction of motion.

[0012] Some embodiments of the present invention allow for particularly simple integration of the present invention into existing technology. These embodiments are feasible when a masking function in an image processing chain applies a masking threshold that can be configured independently for each image region. According to these embodiments, the masking function is then adapted to maintain or extend the privacy mask by reducing the masking threshold in any image region to which the privacy mask was applied upon detected abrupt changes. This allows the present invention to be implemented with little or no costly and time-consuming modification of the image processing chain.

[0013] A further group of embodiments focuses on the length of the period during which the privacy mask is maintained or expanded. The period may be described as a recovery period for the video camera or an auto-exposure loop associated with the video camera. In one embodiment, the length of the period is predetermined (and may be set according to certain heuristics, described below). Alternatively, the period may be determined based on the rate of change of the average luminance, luminance histogram, luminance variance, exposure mismatch, or a control variable related to the exposure of the video camera, where the rate of change may be estimated over successive frames of the video stream. Alternatively, the period may be determined based on the magnitude of the exposure mismatch when an abrupt change in luminance is detected. Alternatively, the period may be determined based on how much the average luminance deviates from the setpoint average luminance when an abrupt change is detected.

[0014] Yet another embodiment provides a stopping criterion for the period. More precisely, the period may be interrupted based on a determination that the exposure mismatch (or its absolute value) has returned below a threshold. Alternatively or additionally, the period may be interrupted when the detection score in an image region rises to a level at which the masking function would apply a privacy mask to the same image region as the detected abrupt change; this behavior may be perceived as evidence that the video camera or auto-exposure loop has recovered, and therefore the object detection algorithm is outputting a detection score of the expected magnitude. At least some of these start and stop criteria can be robustly automated, and they can also be easily fine-tuned by adjusting thresholds, etc.

[0015] In a second aspect, there is provided a controller configured to interact with an image processing chain of the type outlined above. To this effect, the controller has a processing circuit and a memory. The controller is configured to detect abrupt changes in the luminance of a scene being captured by the video camera over a period of time while the video camera is capturing the scene, and to maintain or expand an applied privacy mask at the time of the detected abrupt change. This second aspect of the invention generally shares the effects and advantages of the first aspect and can be realized with a corresponding degree of technical variation.

[0016] The present invention further relates to a computer program comprising instructions for causing a computer, in particular a controller, to carry out the above-mentioned method. The computer program may be stored or distributed on a data carrier. As used herein, a "data carrier" may be a transitory data carrier, such as a modulated electromagnetic or light wave, or a non-transitory data carrier. Non-transitory data carriers include volatile and non-volatile memories, such as permanent and non-permanent storage media of the magnetic, optical, or solid-state type. Also within the scope of "data carrier," such memories may be fixedly mounted or portable.

[0017] In general, all terms used in the claims should be interpreted according to their ordinary meaning in the art, unless expressly defined otherwise herein. All references to "an element, apparatus, component, means, step, etc." should be interpreted broadly as referring to at least one example of the element, apparatus, component, means, step, etc., unless expressly specified otherwise. The steps of any method disclosed herein do not necessarily have to be performed in the exact order described, unless expressly specified otherwise.

[0018] Aspects and embodiments of the present invention will now be described, by way of example, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 illustrates a video camera having an integrated image processing chain and a controller configured to interact with the image processing chain in a manner that ensures privacy masking through episodes of rapid brightness fluctuations. [Figure 2] 1 is a flowchart of a method according to an embodiment of the present disclosure. [Figure 3] FIG. 1 illustrates a video camera having an external image processing chain and a controller configured to interact with the image processing chain in a manner that ensures privacy masking through episodes of rapid brightness fluctuations. [Figure 4] This figure shows three privacy-masked video frames acquired just before an abrupt change in luminance (t=0) and at two subsequent time points (t=1, 2), illustrating how gradually expanding the privacy mask can protect the masking of moving objects. [Figure 5] FIG. 10 illustrates the time evolution of an exposure mismatch indicator. DETAILED DESCRIPTION OF THE INVENTION

[0020] Aspects of the present disclosure will now be described more fully with reference to the accompanying drawings, in which specific embodiments of the invention are shown. These aspects may, however, be embodied in many different forms and should not be construed as limiting; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of all aspects of the present invention to those skilled in the art. Like numbers refer to like elements throughout the specification.

[0021] At a high level, Figure 1 illustrates a video camera 120 configured to output a video stream 140. The video camera 120 may be a digital video camera 120 adapted for monitoring or surveillance applications. The video camera 120 may be fixedly mounted. In the video camera 120, an image sensor 121 provides a raw video stream to an image processing chain 110. The image processing chain 110 may include an object detection algorithm 114 configured to output a frame-by-frame detection score for each image region. The image processing chain 110 may further include a masking function 116 that applies a privacy mask 141 depending on the detection score.

[0022] The object detection algorithm 114 may be configured or trained to detect objects requiring masking, such as human faces, keypads, or vehicle license plates. The detection score D(i) of the object detection algorithm 114 for an image region i may be a confidence level or probability, which can be interpreted as the likelihood that an object of the detected type is present in the image region. Alternatively, the detection score may refer to an arbitrary scale [0, 1], whose endpoints may conceptually convey levels such as "no indication that an object is present" (0) and "highest certainty that an object is present" (1). The detection score is provided as a table or map of values ​​that associates each image region with a detection score value. Object detection algorithms that provide such detection scores are well known and commercially available. The masking function 116 may perform a thresholding operation, for example, such that a privacy mask is applied to image regions whose detection score is equal to or greater than a masking threshold D. A simplified example with only nine image regions per frame and a constant masking threshold D = 0.70 is shown in Table 1. The masking applied to TIFF2023055654000002.tif73170 image regions 5, 7, and 8 may include removing visual features from the video image. For example, the areas corresponding to image regions 5, 7, and 8 may be cropped or trimmed, and the image data may be permanently removed before the video stream exits video camera 120. Alternatively, the same image regions may be overlaid with a static masking pattern, with the original image data replaced by the masking pattern. Alternatively, the image data in the image regions may be blurred, pixelated, or otherwise rendered unintelligible. Pixelation may involve dividing the area into smaller blocks and replacing the image data in each block with a single value, such as the average of the pixels in the block or one of the pixel values ​​in the block. Yet another option is for masking function 116 to apply the privacy mask without modifying the video stream itself, but instead append (mandatory) masking instructions to the video stream to be executed by a playback application at the receiving end. Thus, the raw video stream provided by image sensor 121 may contain other or different information than the processed video stream 140 output by video camera 120 .

[0023] The image processing chain 110 may optionally include an auto-exposure (AE) algorithm 112 configured to reduce exposure mismatch by gradually increasing and decreasing a control variable related to the video camera's exposure. A video frame captured by the video camera 120 has an exposure mismatch if it does not correspond to a desired (or target or setpoint) exposure level. The exposure mismatch may be represented as an indicator ΔE, whose magnitude reflects the severity of the mismatch and whose sign corresponds to overexposure or underexposure. The exposure-related control variable may be exposure time, image sensor gain, or a similar variable if the second option is used. A typical range for exposure time is 1 to 30 ms. The AE algorithm 110 may include a closed control loop, for example, operating as a proportional (P) controller. The P controller may be stateful (adaptive) or stateless. The control loop may further include an integral (I) term or a derivative (D) term, or both. The I and D terms may refer to a history (e.g., a sliding window) of the exposure mismatch indicator ΔE. In addition to adjusting exposure-related control variables, the AE algorithm 112 may apply compensation to the raw video stream in order to brighten underexposed frames and darken overexposed frames. Depending on the characteristics of the object detection algorithm 114, the video stream can be provided to the object detection algorithm 114 either before the compensation process of the AE algorithm 112 or after the compensation process as shown in FIG.

[0024] 1 is a controller 130 configured to interact with the image processing chain in a manner that ensures privacy masking through episodes of rapid luminance fluctuations. The primary functionality of the controller 130 is to analyze the video stream to detect rapid changes in luminance, and, upon a positive detection, to maintain or extend over a period of time the privacy mask that was applied at the time of the detected rapid change.

[0025] In the example shown in FIG. 1 , the controller 130 performs detection on the raw video stream provided by the image sensor 121. This allows for a highly responsive detection of brightness fluctuations. Another possible option is to perform detection on the video stream immediately downstream of the AE algorithm 112, i.e., after a compensation process aimed at normalizing brightness, if any, in the video image. The exposure-compensated video stream corresponds to the input data for the object detection algorithm 114. Therefore, detecting brightness changes based on the exposure-compensated video stream can avoid certain false positives if the AE algorithm 112 successfully compensates for some of the sudden brightness fluctuations occurring in the scene. In other words, small or otherwise harmless brightness fluctuations that do not destabilize the object detection algorithm 114 do not need to be dynamically managed.

[0026] 1 illustrates two alternative options that controller 130 may use to achieve maintaining or expanding the privacy mask. Controller 130 may reconfigure masking function 116 (upper horizontal arrow), controller 130 may process the video stream (lower horizontal arrow) before it is output from video camera 120, or a combination of these. Within the second option, if controller 130 processes the video stream to maintain or expand the privacy mask, controller 130 does not need to be authorized to reconfigure or otherwise act on masking function 116, but controller 130 may have read access to the output or input data or configuration parameters of masking function 116 so that controller 130 can determine or reconstruct the applied privacy mask at the time of a detected abrupt change.

[0027] To achieve the above-described functionality, the controller 130 may include input and output interfaces (not shown), a processing circuit 131, and a memory 132. As shown in FIG. 1 , if the processing chain 110 includes the AE algorithm 112, the controller 130 may be configured to receive data from the AE algorithm 112, such as the exposure mismatch ΔE or a control variable related to exposure. It is noted that FIG. 1 , like FIG. 3 , illustrates the controller 130 and other components in block diagram form, with the blocks primarily reflecting the functions or activities performed. The physical resources on which these functions or activities are realized need not be organized or structured as suggested by the blocks in the diagram. In practice, all or most of the blocks visible in FIGS. 1 and 3 may be implemented as software code executing on general processing resources.

[0028] In the example shown in FIG. 1, the image processing chain 110 is integrated into the video camera 120 so that the video stream does not leave the video camera 120 until privacy masking is applied, which protects against unauthorized parties gaining access to the video stream in its condition upstream of the masking, such as the raw video stream.

[0029] A possible alternative architecture is shown in Figure 3, in which the image processing chain 110 and the controller 130 are external to the video camera 120. As suggested by their dashed outlines, the image processing chain 110 and the controller 130 do not need to correspond to physical units but can be realized in distributed and / or networked (cloud) processing resources that have access to the video stream and / or are authorized to reconfigure or inspect the masking function 116. In particular, the image processing chain 110 and the controller 130 can be executed by an external server, such as a video management system. The architecture according to Figure 3 may optionally include a data connection from the AE algorithm 112 of the processing chain 110 to the controller 130, similar to that shown in Figure 1.

[0030] 1 and 3, a hybrid structure is possible as well, in which at least some components of image processing chain 110 are internal components of video camera 120 and controller 130 is external. For example, AE algorithm 112 may be an internal component of video camera 120. To then make exposure mismatch ΔE and / or exposure-related control variables available to controller 130, video camera 120 may incorporate this information as metadata in the video stream or transmit it on a separate frame-tagged (frame-stamped) channel. This is possible in the Exif image file format, which allows the exposure setting for a frame to be read.

[0031] A further alternative architecture is one in which the masking function 116 applies the privacy mask without modifying the video stream itself, but provides the video stream with masking instructions to be executed by the video playback application. In this setup, the action of the controller 130 may be to modify or replace these masking instructions so that the applied privacy mask is maintained or extended over a period of time upon a detected abrupt change.

[0032] 2, a method 200 for interacting with the image processing chain 110 will now be described. The method 200 may be implemented in a controller 130 of the type illustrated in FIGS.

[0033] In detection step 212, which should be performed while video camera 120 is capturing a scene, controller 130 attempts to detect abrupt changes in the scene's luminance. In a particular example, this includes abrupt changes in luminance that affect only a portion of the scene. Controller 130's detection may include monitoring one or more of the following quantities for successive frames of the video stream: average luminance, luminance histogram, luminance variance (e.g., variance across each video frame), exposure mismatch ΔE, and control variables related to the exposure of video camera 120. These quantities may refer to the raw video stream or processed versions, such as those read at a point further downstream from image sensor 121. Monitoring may be performed separately for different subareas of a video frame. Abrupt changes in luminance may be deemed to exist if the rate of change of the monitored quantity exceeds a threshold, such that the monitored quantity as a whole has changed by more than a threshold increment from the previous frame. Alternatively, abrupt changes may be detected based on the criterion that the monitored quantity has changed by more than a threshold increment over the most recent n0 frames (n0≧2).

[0034] Such monitoring of exposure mismatch ΔE (a scalar-valued quantity) is shown in Figure 5. The beginning of a period of length T during which the privacy mask should be maintained or expanded is marked by a rapid increase in exposure mismatch ΔE at t=0.

[0035] The rate of change of the luminance histogram is the vector norm (L p ,l p ) or probability norms (Bhattacharyya distance, Kullback-Leibler distance, and many further options). Alternatively, the rate of change of the luminance histogram may be observed by tracking changes in the histogram mean or histogram variance (these statistics refer to the frequency of each bin in the histogram), or by tracking the movement of a selected reference point on the histogram.

[0036] The exposure-related control variable of the video camera 120 may be exposure time, image sensor gain, or a similar variable. As explained above, the exposure-related control variable may be adjusted in a closed loop by the AE algorithm 112, whose task is to monitor exposure mismatches and adjust the exposure-related control variable accordingly. As a result, ignoring transients, higher scene brightness ultimately leads to shorter exposure times, and vice versa. Thus, the option of monitoring the exposure-related control variable provides a useful indirect approach to detecting sudden changes in brightness in the scene, thereby avoiding duplicating the existing brightness monitoring of the AE algorithm 112. Advantageously, in a processing chain 110 in which the control loop of the AE algorithm 112 includes a derivative (D) term to specifically counteract sudden changes, the adjusted exposure-related control variable may reflect the underlying brightness fluctuations with specific emphasis.

[0037] In a refinement of step 212, abrupt changes in luminance are detected separately for different blocks (or portions) of the image. More precisely, it can be concluded that a first block of a video image experiences such an abrupt change, while a second and third block do not. Therefore, the action taken according to this method 200 in response to the abrupt change can be limited to the first block, as described further below. This is particularly relevant in high dynamic range (HDR) scenes where multiple exposures may be used.

[0038] In conjunction with the detection step 212, an optional period determination step 214 may be performed in which the length of time T for which the privacy mask is maintained or expanded is determined based on the rate of change or magnitude of deviation for one or more of the quantities monitored in the detection step 212. Specifically, the determination may depend on: - the rate of change of the average luminance, luminance histogram, luminance variance, exposure mismatch ΔE or control variables related to the exposure of the video camera for successive frames of the video stream; - the magnitude of the exposure mismatch ΔE at the detected abrupt change, and / or - The magnitude of deviation of the mean brightness from the setpoint mean brightness during the detected abrupt change. The setpoint average brightness may correspond to a desired average brightness and may form part of the configuration parameters of the AE algorithm 112.

[0039] In embodiments where the maintain / extend period is not determined during execution of method 200, a predetermined length T may be used. One example heuristic for setting the length of the predetermined period is to associate the length of the predetermined period with a dynamic property of the AE algorithm 112, such as the duration of an impulse response. The duration of the impulse response relates to how quickly the AE algorithm 112 reacts to an exposure mismatch of a given magnitude, and the duration of the impulse response may in turn depend on the configured values ​​of various control gains in the AE algorithm 112. The duration of the impulse response may refer to the period during which the absolute value of the impulse response differs from zero by at least a positive threshold ε>0, i.e., ignoring the long tail. With this understanding, because a relatively long duration of the impulse response may indicate that the image processing chain 110 requires relatively more time to recover from a sudden change in scene luminance, the privacy mask applied during the detected sudden change should be maintained (or extended) for a relatively long time to allow the object detection algorithm 114 to stabilize. Conversely, for similar reasons, if the impulse response has a relatively short duration, the maintained privacy mask may be needed for a relatively short period of time.

[0040] In a maintain / extend step 216, which is performed while the video camera 120 is still capturing substantially the same scene, one or more actions are taken to ensure that the privacy mask applied at the time of the detected abrupt change remains applied or is extended for the period.

[0041] As described above, such action may include processing the video stream. If masking function 116 is bypassed, processing may include adding a complete privacy mask that covers at least the same image area as at the time of the detected abrupt change. If alternative step 216 involves processing the video stream at a point downstream of masking function 116, processing may include evaluating whether the privacy mask applied by masking function 116 still covers at least the same image area as at the time of the detected abrupt change. If not, privacy masking is added to the image area where there was no privacy mask.

[0042] Alternatively, the action of step 216 may include modifying the settings of the masking function 116. Specifically, assuming that the masking function 116 uses a masking threshold D0=D0(i), which may be independently configurable for each image region i of the video frame F, the masking function 116 may be adapted to maintain or expand the privacy mask by reducing the masking threshold in any image region to which the privacy mask was applied at the time of the detected abrupt change. If a privacy mask was applied to image region i∈I (I⊆F) at the time of the detected change, the reduction in the masking threshold may correspond to an assignment D0(i)←α for all i∈I, where α is a small constant value such as −∞ or 0, or a minimum specified value for the detection score. The assignment may be The masking threshold D0(i) remains unchanged for TIFF2023055654000003.tif7170. In this way, the threshold setting operation in masking function 116 applies masking in all image regions i∈I, regardless of the detection score D(i) received from object detection algorithm 114.

[0043] To illustrate how the settings of the masking function 116 can be modified to implement step 216, refer to Table 1. Table 1 reflects the situation just before an abrupt change in luminance occurs. As Table 1 shows, a privacy mask was applied to image regions in the set I={5,7,8}. Until the object detection algorithm 114 recovers from the abrupt change, the object detection algorithm 114 outputs a reduced detection score. To ensure that the privacy mask is still applied, a small constant value α (e.g., 0.00 or 0.02) is assigned to the masking threshold for all image regions in I, as shown in Table 2. TIFF2023055654000004.tif68170

[0044] As outlined above, when the detection step 212 is implemented in a block-by-block manner, it may be advantageous to apply the masking / dilation step 216 only to blocks in which abrupt changes in luminance have been detected. In particular, the masking threshold of the masking function 116 only needs to be reduced in image regions that overlap those blocks.

[0045] The masking / expanding step 216 may be performed until the time period has elapsed, i.e. for T time units. In some embodiments, the method 200 further comprises a step 218 of event-based interruption of the time period. More precisely, - Event 1: Exposure mismatch returns below the threshold |ΔE|≦β Event 2: The detection score D(i) in an image region increases to a level such that the masking function applies a privacy mask in the same image region as the detected abrupt change. and / or - Event 3: The rate of change of the average luminance, luminance histogram, luminance variance, exposure mismatch ΔE, or a control variable related to the video camera exposure for successive frames of the video stream falls below a threshold value. If any of the above is determined, step 216 is aborted. At this point, a new cycle 212 of detecting sudden changes in luminance in the scene can begin. For Event 1, the value of threshold β≧0 can be set by trial and error. More precisely, the video camera 120, when configured with different candidate values ​​β1, β2, ..., can be subjected to rapid luminance fluctuations while the operator monitors whether any undesired unmasking of faces, keypads, license plates, etc. occurs. The maximum candidate value that achieves this can be selected. Event 2 can be formulated as the following set of inequalities: ∀i∈I, D(i)≧D0(i) where I ⊆ F is the set of image regions to which privacy masks were applied at the time of the detected abrupt change. The set of inequalities can be evaluated collectively (all must be satisfied for normal operation to resume) or one by one. In a second option, normal operation may be resumed in image region i as soon as a local increase in the detection score is found to reach a threshold level. In a further variation of this embodiment, event 2 may be relaxed to a criterion such that D(i) ≧ D0(i) is satisfied for at least a certain percentage of the image regions in I, for example, 80% or 90%. The percentage may correspond to the degree of temporary unmasking that is considered acceptable in familiar use cases.

[0046] Some embodiments of the method 200 are particularly adapted to handle moving objects in a scene. More precisely, if it is detected (210) that the scene contains at least one moving object to which a privacy mask is applied during an abrupt change in brightness, the privacy mask is expanded around the moving object. The moving object detection 210 can follow an indirect approach, where the center of the privacy mask (or the centers of disjoint components of the privacy mask) is tracked over time. This eliminates the need to apply dedicated motion detection in addition to the object detection algorithm 114.

[0047] Because the object detection algorithm 114 cannot be assumed to provide a useful detection score immediately after an abrupt change in brightness, a privacy mask may be extended around the last known position of the moving object in the video frame. This may be achieved in a position-agnostic manner as follows: Again, it is assumed that the privacy mask has been applied in an image region of index set I. The open neighborhood of I is defined as follows, for any r>0: I r = {j|For some i∈I, dist(j,i) <r} where dist(·,·) is a distance function that represents the distance between two image regions. Recalling Tables 1 and 2, which refer to a simplified conceptual example, the total number of image regions in a video frame is at least an order of magnitude less than what would be a suitable masking granularity for a commercial monitoring video camera. For example, the image regions may be individual pixels in the frame, or square groups of 4, 16, 25, etc. pixels, respectively. If I refers only to moving objects, then operation 216 of expanding the privacy mask may be performed by expanding the privacy mask to the neighborhood I r This can correspond to applying masking in all image regions of i∈I. This can be achieved, for example, by reducing the masking threshold around moving objects. Thus, specifically, a small constant value α is applied to all i∈I r The radius r may be a predetermined constant r=r0. Alternatively, the privacy mask expansion is gradual over time. If t=0 denotes the time when an abrupt change in luminance occurs, the method 200 calculates the set I r(t) The region within may be masked, and r(t) is a non-decreasing function. For example, an affine function may be used. r(t)=at+b where a>0 and b≧0.

[0048] In a further development of this embodiment, the method 200 further comprises estimating 210.1 the velocity |ν| of the moving object at the abrupt change and making the velocity at which the privacy mask is expanded linearly or non-linearly related to the velocity estimation. More precisely, the privacy mask is expanded in the neighborhood I r(t) When applied to , the growth coefficient a of the affine function r(t) may be related to |ν|. The relationship may be linear or affine. With ζ, η>0, a=ζ|ν|+η can be used to accurately capture the dual contributions from both the position uncertainty itself, which grows over time, and the velocity of migration |ν|.

[0049] In a further development of this embodiment, velocity estimation 210.1 further includes estimating the direction of motion of the moving object during the abrupt change, thereby making available an estimate of the motion vector v. This estimate may be used for two purposes. On the one hand, the expansion of the privacy mask can be substantially limited to the estimated direction of motion, for example by adding image areas to the front edge of the privacy mask but not to the rear edge. On the other hand, the privacy mask can be translated in the estimated direction of motion, in other words, the privacy mask is expanded in the direction of motion and reduced in the opposite direction. Each option tends to track the moving object with reasonable accuracy over a period of time, without obstructing the field of view by unnecessarily expanding the privacy mask.

[0050] 4 shows the gradual expansion of the privacy mask 141 caused by the detected face of a walking person. The width (horizontal extent of the video frame) of the privacy mask 141 at the time of the sudden change in luminance (t=0) is w0 units. During the period in which step 216 is performed, the width of the privacy mask 141 is gradually increased at t=1 and t=2 according to the function: w(t)=w0+w1t where w1>0 is a constant. The privacy mask 141 remains centered at the same image point, which corresponds approximately to the position of the walking person's face at the time of the abrupt change. As shown, the gradual expansion of the privacy mask is r This can be achieved without using construction.

[0051] According to the first further development mentioned above, the growth factor w1 is related to the estimated velocity |ν| of the walking person. Velocity refers to the movement as seen in a video image and may be expressed in pixels per second. According to the second further development mentioned above, the direction of movement of the person is estimated and found to correspond approximately to the Cartesian vector (1,0). Under this second development (not shown in FIG. 4), it is possible to expand the privacy mask 141 in width direction without expanding it in height direction.

[0052] Aspects of the present disclosure have been described above primarily with reference to certain embodiments, however, as will be readily recognized by those skilled in the art, other embodiments than those disclosed above are equally possible within the scope of the present invention as defined by the appended claims. [Explanation of symbols]

[0053] 110 Image Processing Chain 112 Auto Exposure (AE) Algorithm 114 Object Detection Algorithm 116 Masking function 120 Video Camera 121 Image Sensor 130 Controller 131 Processing circuit 132 memory 140 video streams 141 Privacy Mask 200 ways 210 Detection Step 210.1 Speed ​​Estimation Steps 212 Detection Step 214 Period determination step 216 Maintain / Extend Steps 218 Interrupt Step

Claims

1. A method (200) for interacting with an image processing chain (110) configured to process a video stream captured by a video camera (120), said method comprising: an object detection algorithm (114) that outputs a frame-by-frame detection score for each image region; a masking function (116) that applies a privacy mask (141) depending on the detection score; Including, the masking function is configured to apply the privacy mask to image regions where the detection score exceeds a configurable masking threshold for each image region; While the video camera is capturing the scene, Detecting (212) abrupt changes in the scene brightness; maintaining or expanding the privacy mask over a period of time (216), including reducing the masking threshold of the masking function in areas of the image to which the privacy mask is applied upon the detected abrupt change; is executed, method.

2. The detecting (212) of abrupt changes may include detecting, for successive frames of the video stream: Average brightness, Brightness histogram, Luminance dispersion, exposure mismatch, a control variable related to the exposure of said video camera; monitoring one of the amounts of The method of claim 1.

3. detecting (210) that the scene contains a moving object to which the privacy mask is applied; the privacy mask is expanded around the moving object; The method of claim 1.

4. The method of claim 3 , wherein the privacy mask is expanded around the moving object gradually over time.

5. estimating the velocity of the moving object (210.1); the privacy mask is expanded around the moving object in proportion to the estimated velocity; The method of claim 3.

6. the estimating of the velocity (210.1) comprises estimating the direction of motion of the moving object; the extension of the privacy mask is substantially limited to the estimated direction of movement and / or the privacy mask is translated in the estimated direction of movement; The method of claim 5.

7. The method of claim 3 , further comprising reducing the masking threshold around the moving object.

8. The method of claim 1 , wherein the length of the period is predetermined.

9. the image processing chain (110) further includes an auto-exposure algorithm (112) configured to reduce exposure mismatch by gradually increasing and decreasing an exposure time control variable of the video camera; the length of the period is related to the dynamic nature of the auto-exposure algorithm; The method of claim 8.

10. the rate of change of the average luminance, luminance histogram, luminance variance, exposure mismatch, or a control variable related to the exposure of the video camera for successive frames of the video stream; the magnitude of the exposure mismatch at the detected abrupt change; and the magnitude of deviation of the average brightness from a setpoint average brightness at the time of the detected abrupt change; 2. The method of claim 1, further comprising determining (214) the length of the period based on:

11. The exposure mismatch returns below the threshold, and the detection score in an image region increases to a level such that the masking function applies the privacy mask in the same image region as the detected abrupt change; 2. The method of claim 1, further comprising: interrupting (218) the period in response to one or more determinations of:

12. The method of claim 1 , wherein the video camera (120) is fixedly mounted.

13. A non-transitory computer-readable storage medium storing instructions for performing the method of any one of claims 1 to 12 when executed on a device having processing capabilities.

14. a controller (130) configured to interact with an image processing chain (110) configured to process a video stream captured by a video camera (120), an object detector that analyzes a plurality of image regions and outputs a frame-by-frame detection score for each image region; an object masking device for applying a privacy mask (141) according to the detection score; Including, the object masking unit is further configured to apply the privacy mask to image regions where the detection score exceeds a configurable masking threshold for each image region; the controller comprises a processing circuit (131) and a memory (132); detecting abrupt changes in luminance of a scene being captured by the video camera; maintaining or expanding the privacy mask over the period the video camera is imaging the scene, including reducing the masking threshold in image regions to which the privacy mask is applied upon the detected abrupt change. It was configured as follows: controller.

Images