A Reinforcement Learning-Based System for Tuning Camera Parameters to Improve Analysis

Abstract

A method is presented for automatically modifying camera parameters to improve video analysis accuracy through continuous changes in environmental conditions. The method includes ingesting video streams from multiple cameras, performing video analysis tasks defined as analysis units (AUs) on the video streams, applying image processing to the video streams to obtain processed frames, filtering the processed frames through a filter to discard low-quality frames, and dynamically fine-tuning parameters of the multiple cameras. The fine-tuning includes passing the filtered frames to an AU-specific proxy quality evaluator, automatically fine-tuning the parameters of the multiple cameras using a State-Action-Reward-State-Action (SARSA) reinforcement learning computation, and applying a new policy for an agent to take actions and learn to maximize rewards based on the reinforcement computation.

Description

[Technical field]

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 196,399, filed June 3, 2021, and U.S. Patent Application No. 17 / 825,519, filed May 26, 2022, the disclosures of both of which are incorporated herein in their entireties.

[0002] The present invention relates to video analysis, and more particularly to a reinforcement learning based system for adjusting camera parameters to improve analysis. [Background technology]

[0003] Video analytics has become commonplace over the past decade thanks to advances in computer vision algorithms, deep learning, edge computing, 5G, etc. This market has gained further momentum in the era of COVID-19, as government agencies see value in analyzing footage to address emerging COVID-19-specific applications such as crowd control, social distancing, and contact tracing. The deployment of ubiquitous edge cameras across cities makes it possible to perform various analytical tasks on the footage. However, these videos are directly affected by the environment in which they are located, which is ever-changing. Failure to account for and adapt to these changes can result in reduced accuracy of analytical tasks. Summary of the Invention

[0004] A method is presented for automatically modifying camera parameters to improve video analysis accuracy through continuous changes in environmental conditions. The method includes ingesting video streams from multiple cameras, performing video analysis tasks on the video streams, defined as analysis units (AUs), applying image processing to the video streams to obtain processed frames, filtering the processed frames through a filter to discard low-quality frames, and dynamically fine-tuning parameters of the multiple cameras. The fine-tuning includes passing the filtered frames to an AU-specific proxy quality evaluator, automatically fine-tuning the parameters of the multiple cameras using a State-Action-Reward-State-Action (SARSA) reinforcement learning (RL) computation, and applying a new policy for an agent to take actions and learn to maximize reward based on the reinforcement computation.

[0005] A non-transitory computer-readable recording medium is presented that includes a computer-readable program for automatically modifying camera parameters to improve video analysis accuracy through continuous changes in environmental conditions. When the computer-readable program is executed on a computer, it causes the computer to capture video streams from multiple cameras, perform video analysis tasks defined as analysis units (AUs) on the video streams, apply image processing to the video streams to obtain processed frames, filter the processed frames through a filter to discard low quality frames, and dynamically fine-tune parameters of the multiple cameras. The fine-tuning includes passing the filtered frames to an AU-specific proxy quality evaluator, automatically fine-tuning the parameters of the multiple cameras using a State-Action-Reward-State-Action (SARSA) reinforcement learning (RL) computation, and applying a new policy for an agent to take actions and learn to maximize rewards based on the reinforcement computation.

[0006] A system for automatically modifying camera parameters to improve video analysis accuracy through continuous changes in environmental conditions is presented. The system has a memory and one or more processors in communication with the memory configured to ingest video streams from multiple cameras, perform video analysis tasks defined as analysis units (AUs) on the video streams, apply image processing to the video streams to obtain processed frames, filter the processed frames through a filter to discard low quality frames, and dynamically fine-tune parameters of the multiple cameras. The fine-tuning includes passing the filtered frames to an AU-specific proxy quality evaluator, automatically fine-tuning the parameters of the multiple cameras using a State-Action-Reward-State-Action (SARSA) reinforcement learning (RL) computation, and applying a new policy for an agent to take actions and learn to maximize reward based on the reinforcement computation.

[0007] These and other features and advantages will become apparent from the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings.

[0008] In the present disclosure, preferred embodiments are described in detail with reference to the following drawings, as described below. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block / flow diagram of an exemplary control loop for modifying camera parameters, in accordance with an embodiment of the present invention.

[0010] [Diagram 2] FIG. 2 is a block / flow diagram of an exemplary control loop for modifying virtual camera parameters, in accordance with an embodiment of the present invention.

[0011] [Diagram 3]FIG. 3 is an exemplary practical application for automatically changing camera parameters to improve video analysis accuracy through continuously changing environmental conditions, according to an embodiment of the present invention.

[0012] [Figure 4] FIG. 4 is an exemplary processing system for automatically modifying camera parameters to improve video analysis accuracy through continually changing environmental conditions, in accordance with an embodiment of the present invention.

[0013] [Diagram 5] FIG. 5 is a block / flow diagram of an exemplary method for automatically modifying camera parameters to improve video analysis accuracy through continually changing environmental conditions, in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Video analytics has become very commonplace and widespread over the past decade, owing to advancements in computer vision algorithms, artificial intelligence, deep learning and machine learning, growth in the Internet of Things (IoT) and edge computing, coupled with the availability of 5G, which allows for low latency and high bandwidth communications. These factors were already driving the overall video analytics market size growth in various sectors, including surveillance, traffic monitoring, retail analytics, healthcare, banking, etc., but COVID-19 has further accelerated the growth.

[0015] Specifically, ensuring social distancing, managing lockdowns and containment zones by government agencies, fever screening, limiting crowds and total number of people at any given time within a premises, recognizing individuals wearing masks, identifying demographic profiles, enabling contactless transactions, payments and boarding procedures etc. are proving to be new demands and video analytics is playing a vital role in aiding the day-to-day work of authorities in dealing with these emerging COVID-19 specific scenarios.

[0016] Cameras are deployed at a city-wide scale all over the world, and different kinds of video analytics tasks are performed on these footage. These video analytics tasks are called Analytics Units (AUs). Depending on the requirements of the AU, the cameras are deployed either indoors or outdoors. For example, for traffic analysis, video cameras are deployed outdoors facing roads or highways, while for contactless boarding procedures, cameras are deployed indoors facing the entrance of the boarding gate. For the same task, e.g., video surveillance, cameras are deployed indoors (e.g., hallways in buildings) as well as outdoors (e.g., parking lots). Different AUs therefore require video cameras deployed in different locations, each capturing a different scene. In particular, conditions may provide natural light from the sun, artificial lighting, or a combination of both. This can result in bright or dark scenes, and changes over time. For example, outdoor conditions vary depending on weather conditions (cloudy, sunny, rainy, foggy, etc.) and time of day (dawn, morning, afternoon, evening, dusk, twilight, night, etc.), while indoor conditions vary depending on the indoor placement location and the amount of artificial and natural light in the area.

[0017] A particular direction of lighting changes the direction of the cast shadows, and therefore the way the camera sees the scene. All these dynamically changing environmental conditions affect the accuracy of the AU. For real-time response, the accuracy depends on the last observed video frame, which is directly affected by the conditions observed by the camera at that time.

[0018] The final image produced by any camera depends on the type of camera lens, camera module and sensor, as well as the entire Image Signal Processing (ISP) pipeline used in the camera system. The ISP plays a key role in this, converting the captured raw data into the final compressed image. As part of this, several configuration parameters, called knobs, are exposed that adjust the final image quality. The exemplary method focuses on a subset of these knobs. Although camera suppliers provide default settings for various knobs, these default settings do not come out of the box and do not provide good image quality in various placement conditions. These provide guidance in manually adjusting multiple knobs and calibrating for a specific placement condition depending on the location. However, this manual adjustment and calibration assumes that the conditions at that specific location remain approximately the same. As mentioned above, conditions keep changing, so the initial calibration may not work well for new conditions.

[0019] Therefore, there is a need to continuously modify these camera parameters so that the ISP pipeline can generate high-quality images suitable for video analytics tasks. Moreover, this needs to be done for each camera at a city-wide scale, which is impractical for humans to do. CamTuner is proposed herein to address this need and automatically modify camera parameters (at a city-wide scale) to improve video analytics accuracy. An exemplary method exploits the fact that environmental changes do not occur suddenly, but gradually over a period of time, allowing CamTuner to constantly monitor conditions and dynamically modify and fine-tune to gradually changing conditions.

[0020] Exemplary embodiments of the present invention empirically show that different AUs undergo different levels of degradation if the camera parameters are not changed. Thus, the exemplary method shows that there are better settings of camera parameters that can improve the performance of the AU. Exemplary embodiments of the present invention introduce a new mechanism for finding digital equivalents of physical camera parameters, called digital twins, which allows analyzing the performance of the AU under different camera settings. Exemplary embodiments of the present invention verify that for an AU, different environmental conditions result in different optimal camera settings, thereby raising the need for automatic and dynamic adjustment. Exemplary embodiments of the present invention introduce a State-Action-Reward-State-Action (SARSA) Reinforcement Learning (RL) algorithm (or computation) to dynamically adapt to changing environmental conditions, and as part of this, the exemplary method introduces a new policy for an agent to take actions and learn to maximize rewards.

[0021] Exemplary embodiments of the present invention propose a Virtual Camera (VC) simulator that allows the system to apply different environmental characteristics, to study the impact of different camera settings on the exact same scene, and to train and test RL algorithms for automatic camera parameter adjustment. At runtime, RL requires continuous and automatic rewards for every action performed. It has been observed that the SOTA AQuA filter cannot provide a fine-grained quality assessment with respect to different AUs. AQuA stands for Analytic Quality Assessment. AQuA protects the accuracy of applications on distorted frames by scoring the distortion level of the frame. AQuA considers the analytical quality of the frame instead of the visual quality by learning a new metric, the opinion score of the classifier, and uses a light CNN-based object-agnostic feature extractor. AQuA accurately scores the distortion level of a frame and generalizes to multiple different deep learning applications.

[0022] To address this, the exemplary method defines an analysis quality (AQ) metric specific to each AU and uses the metric to train an AU-specific AQuA classifier that provides a quality reward for each RL action. Exemplary embodiments of the present invention conduct multiple evaluations that demonstrate the effectiveness of various components used in CamTuner. CamTuner is a key system that adapts to changing environmental conditions by automatically adjusting camera parameters based on the needs of a particular AU, improving the accuracy of the AU.

[0023] While CamTuner focuses on the knobs on the camera, the high level idea applies to sensors that have multiple knobs to be adjusted and changed to adapt to continuously changing environmental conditions. One example is a thermal camera, where knobs such as "object emissivity", "atmospheric transmittance", and "ambient temperature" are disclosed for "temperature compensation". For accurate temperature measurement, these parameters need to be continuously changed based on the actual environmental conditions. And for a PTZ camera, knobs such as "pan", "tilt", and "zoom" need to be continuously changed based on the analysis requirements as the object of interest moves in the scene.

[0024] Environmental changes over the course of a day introduce various distortions in the frames being acquired (e.g., overexposure in sunlight, low-light noise, underexposure at night, etc.). These distortions affect the performance of various AUs and in most cases degrade performance if the camera settings are not changed according to the changing conditions. Due to a lack of publicly available video datasets that capture environmental changes throughout the day from the same camera position, the exemplary method uses privately owned videos taken outdoors, such as at a baseball field, with a single camera from morning to evening.

[0025] With default camera settings, the accuracy of AUs for the same camera feed varies throughout the day due to the presence of different environmental conditions. Specifically, at midday (when there is bright sunlight), faces in the captured frames are overexposed, resulting in lower accuracy than the AU performance for morning videos. However, evening videos introduce low-light noise, which also reduces accuracy. Faces in mid-afternoon video frames are slightly occluded, making them difficult for the face detector to detect, but the lighting conditions throughout the afternoon are good, allowing the person detector AUs to accurately predict bounding boxes, resulting in optimal mAP performance.

[0026] With regard to the impact of camera settings on AUs, since there is at least a best camera setting that can improve all the accuracy observed with the default settings, the exemplary method considers how different combinations of camera parameters affect different images and video snippets captured in different ways in various environmental conditions.

[0027] With respect to the virtual knobs, to be repeatable and consistent, an exemplary method requires that a given scene can be captured and different camera settings can be applied to the same scene. Since the scene keeps changing but the camera has only one setting for the various knobs, it becomes important to find a digital equivalent to the actual physical settings of the camera. This digital equivalent must faithfully produce an image as if it was captured by modifying the physical settings of the camera itself. This digital equivalent is called the camera's digital twin. According to Algorithm 1 described below, the camera digital twin for the four adjustment parameters (brightness, contrast, saturation, sharpness) of the AXIS Q1615 network camera is as follows: [Table 1]

[0028] This digital twin helps to find an equivalence search space in the digital domain, which can be achieved by traversing the entire configuration space of the physical camera. The knob values ​​in the digital domain are treated as reliable representations of the physical knob values ​​only if the images acquired in both domains with the respective knob values ​​provide the maximum SSIM (at least greater than 0.90). The equivalence search space in the digital domain depends on the full-reference Image-Quality assessment (FR-IQA) method used to test the similarity between the digitally transformed camera frames and the actual physical camera frames, such as structural similarity (SSIM), multiscale-structural similarity (MS-SSIM), MSE and PSNR, and the image transformation algorithm used. The extent of these various equivalence search spaces in the digital domain is shown in Table 1. [Table 2]

[0029] Finally, the union of these different ranges obtained by different FR‐IQA methods for a particular tuning parameter is taken as the equivalent digital search space, listed in Table 2 . [Table 3]

[0030] Regarding the impact of changing the virtual knob, we use an equivalent range of parameters in the digital domain to determine how the AU accuracy changes while simultaneously changing one of the four tuning parameters for every image or frame. An exemplary method explores the best possible impact on the AU accuracy by changing a single parameter at a time.

[0031] Varying the saturation affects the analytical accuracy of face detection (i.e., mAP) by 5% for 100 random images from the FiveK raw image dataset. Moreover, by changing a single tuning knob, the face recognition AU recognizes 10 more images than the baseline among 1150 randomly selected celebrity images from the celebA dataset. The liveness detection AU detects whether an image is real or fake (i.e., plays an important role in cashless payments), and by changing the contrast tuning knob, it improves the accuracy of binary classification by 6.8%, correctly classifying 20 more images as real or fake among 280 images from the MSU-MFSD dataset.

[0032] Varying the saturation knob (i.e., keeping the other knobs fixed) improves the mAP of the face detection AU by 7% for private nighttime videos. For video snippets from the Olympic Action dataset, varying contrast improves the mAP of the person detector AU by 9%, while varying saturation improves the mAP of the face detector AU.

[0033] Even a crude approach of tuning a single parameter individually for all images present in the dataset can provide better analysis results. However, since the impact of each camera's tuning knob on the analysis accuracy is inherently non-monotonous and each image or video segment is captured under different environmental conditions and camera settings, a joint tuning of these four image appearance tuning knobs is considered for each still image and video snippet.

[0034] As for still images, each image available in public datasets (e.g., FiveK RAW dataset, celebA, etc.) is a proxy for a video snippet that would have been taken around the time the image was captured. Therefore, each image is captured in different environments and with different camera settings. The optimal parameter settings for the camera correspond to the configuration that provides the best AU-specific analysis quality.

[0035] For detection AUs, an exemplary method uses Efficientdet-v0 as a person detector and RetinaNet as a face detector for 100 randomly selected FiveK raw images. We first manually annotate people and corresponding face bounding boxes for these 100 raw images. Algorithm 2 (reproduced below) shows the optimal tuple search algorithm for either face or person detection AUs, considering both mAP and true positive IoU as analysis quality metrics.

[0036] The exemplary method initially uses the parameter ranges shown in Table 2, but it was observed that the best configurations were near the upper end of the parameter ranges obtained using the digital twin. Therefore, the exemplary method expanded the configuration space so as not to miss another possible best configuration.

[0037] The face detector AU is able to detect faces in 39 more images after tuning with the optimal configuration over the baseline configuration, but for person detection, tuning with the optimal configuration leads to an average mAP improvement of 2.8% and an IoU improvement of 6.7%. Thus, after tuning on more fine-grained analyses such as face detection, we see higher improvements compared to person detection on the same set of images. [Table 4]

[0038] Regarding the face recognition AU, an exemplary method considers a face recognition AU pipeline. Given a database of people with at least one face image associated with it, the AU recognizes faces in the input image as known people (already in the database) or classifies them as unknown using a match score. One image per person from the CelebA dataset is used to build the database, and various augmented versions of another randomly selected image of the same person are used as query images. The analytical quality (AQ) of the face recognition AU is defined as the true positive match score for any image.

[0039] Adjusting the four knobs in combination helps to correctly recognize faces in 41 (13.67%) more images out of 300 unregistered images than the base configuration (i.e., no augmentation). Even the match score for true positive recognition improves by an average of 3.6% (e.g., analysis quality improves by up to 11.6%). Changing the minEyeDistance configuration from 20 to 40 significantly improves recognition, with 20.67% more images being recognized with the optimal augmentation configuration compared to the base configuration.

[0040] Regarding the Facial Keypoint Detection AU, the facial keypoint detection AU detects 68 facial landmarks of the faces present in the images of the 300W dataset. The analysis quality of this AU is defined as the root-mean-square (RMS) distance between the predicted landmarks and the actual landmarks.

[0041] Compared with the baseline configuration, after tuning with the optimal configuration, facial landmarks are detected in more than 11 images out of 230 person images. The analysis quality of the detected images is improved by an average of 31.1% under tuning with the optimal configuration compared to simply using the baseline configuration.

[0042] The variation in optimal parameters across the configuration space implies that images taken in different environments with different camera settings require different attention (i.e., different optimal configurations).

[0043] For video snippets, each video snippet contains up to a few hundred consecutive frames, so that similar environmental conditions and camera settings are exhibited throughout the entire shoot. An optimal configuration of a video snippet should provide optimal analysis quality across all frames of the entire video snippet. An exemplary method uses detection AUs (e.g., face and person detection AUs) on 19 video snippets from the HMDB dataset and 11 video snippets from the Olympics dataset after manually annotating face and person bounding boxes using a cvat tool.

[0044] Table 3 below shows the maximum and average analysis quality improvement achievable after different adjustments for each video snippet. [Table 5]

[0045] Each video segment in the Olympics and HMDB datasets is highly compressed. Therefore, the coarser person detection AUs look for a much larger space, i.e., more pixels in the frame, to detect a person, and can tolerate such compression better than the finer face detection AUs. However, the finer face detections generally look for a much smaller space, i.e., fewer pixels in the frame, to detect a face, but are less likely to improve on small face detections, which suffer from more severe compression artifacts than the person detection AUs shown in Table 3.

[0046] For each still image or video snippet showing different environmental conditions with different camera settings (embedded with different content) in conjunction with different AUs, there is a unique set of parameters that provides the highest AU accuracy. These parameters are not identifiable by humans. Moreover, humans cannot keep changing these parameters as the environment changes. Therefore, automatic tuning of these knobs is necessary to get optimal AU performance under various environments and camera settings.

[0047] A control loop for modifying the tuning knobs of the network camera and improving the analysis quality of a given AU is shown in the system 100 of FIG. 1. Live video feeds are continuously captured by the image sensor 102 of the network camera from the external environment, and various image processing techniques 104 are applied through the ISP pipeline to obtain the final processed frames. Before offloading to edge servers or remote clouds for further analysis using wireless networks (4G or 5G), these processed frames are filtered by a light AQuA filter 106 that helps to discard low quality frames that may cause errors (not suitable for AUs) with a high degree of certainty. This also helps to reduce the bandwidth requirements of the wireless network by early discarding frames that do not contribute to improving the accuracy of the AU or adjusting the camera parameters for better analysis.

[0048] At the edge server, the filtered and offloaded frames pass through different analysis pipelines and fine-grained AU-specific analysis quality evaluators 110 in parallel. An exemplary method can use analysis quality evaluators with joint mAP and IoU metrics for any detection AU, match scores for face recognition AU, and RMS distance between predicted and actual keypoints in keypoint detection. However, these evaluators all require ground truth (or labeled data). This is not possible or practical at runtime unless a human manages a control loop that continuously labels incoming frames for some analysis.

[0049] Instead of generating a ground truth that can be used to determine the AU quality (not possible in practical deployments), the exemplary method introduces a fine-grained proxy quality evaluator 110. The proxy quality evaluator 110 is AU-specific and provides a quality estimate that can be used to determine the quality of the offloaded frames. Based on the AU quality determined by the proxy quality evaluator 110, the camera parameters are automatically adjusted to further improve the AU quality. Camera suppliers provide APIs that allow the camera parameters to be changed programmatically. Typically, parameter changes take a very short time (approximately 150-200 ms), making the present approach suitable for dynamically adjusting camera parameters. The adjustment process continues for the lifetime of the AU. Specifically, the exemplary method utilizes reinforcement learning (RL) techniques to adjust the parameters.

[0050] With respect to camera parameter tuning, to automatically tune the camera parameters, the exemplary method utilizes RL techniques for each individual AU 108. In particular, the exemplary method uses a State-Action-Reward-State-Action (SARSA) RL algorithm 115 or RL computation. In this setting, there is an agent that interacts with the environment, i.e., the state of the environment, by performing various actions. As the agent performs an action, the agent arrives in a new, different environment. Each action has an associated reward or penalty depending on whether the newly achieved state is desirable or not, i.e., whether it leads the agent in a positive or negative direction. Over time, as the agent continues to perform actions and receive rewards and penalties, the agent learns to maximize rewards by taking the correct actions, which ultimately leads the agent in a positive direction.

[0051] SARSA115 does not require labeled data or pre-trained models, it learns and updates its policy based on the actions it performs. Therefore, SARSA115 is a policy-based learning algorithm that requires a clear definition of the environment, i.e., states, actions, and rewards. For the purpose of automatic camera parameter tuning, the following are defined:

[0052] In terms of state, this is a vector containing the camera's current brightness, contrast, sharpness and color parameter values, as well as measurements of brightness, contrast, sharpness and color.

[0053] Regarding actions, the set of actions that the agent can perform is to increase or decrease one of the parameter values: brightness, contrast, sharpness or color, or to perform no action at all.

[0054] With regard to reward, the exemplary method uses the proxy quality evaluator 110 for each AU 108 as a reward function for the SARSA 115. An improvement in the quality estimate is considered a reward, and a decrease in the quality estimate is considered a penalty.

[0055] The reward obtained using the proxy quality evaluator 110 is called the immediate reward. The agent not only considers the immediate reward, but also future rewards that may occur as a result of the current action. Based on this, a value called the Q-value is calculated to perform an action a when in state s. This is denoted as Q(s,a). The value of Q(s,a) is calculated according to the following formula: where α is a constant with a value between 0 and 1. It can be used to control how much importance is given to new information acquired by the agent. A value of 1 gives high importance to new information, while a value of 0 stops the learning phase of the agent. In the initial phase, the operator can assign a higher value (e.g., 0.9). This allows the agent to learn and understand the new environment quickly by placing more importance on the latest information. After the agent has learned and taken in the information of the new environment, the operator can lower the value of α, for example to 0.1, so that the agent starts using the information it has gathered about the new environment.

number

[0056] Similar to α, the above formula includes another constant γ, which has a value between 0 and 1. This constant is called the discount factor and can be used to control the importance the agent gives to long-term rewards. A value of 1 makes the agent care more about long-term rewards, while a value of 0 makes the agent ignore long-term rewards and focus only on immediate rewards. If conditions change very frequently, γ can be assigned a low value (e.g., 0.1) to favor immediate rewards. On the other hand, if conditions change infrequently, a higher value (e.g., 0.9) can be assigned to favor long-term rewards.

[0057] In the SARSA algorithm 115, the algorithm that the agent follows is Algorithm 3 below. [Table 6]

[0058] Here, the agent first observes the environment, i.e., state s, and selects an appropriate action a to be executed in state s. After executing action a, the agent receives an immediate reward (or penalty) r and enters a new state s'. In this new state, the agent's selected action is a' and the associated Q-value is Q(s',a'). The agent uses this to calculate the cumulative reward Q(s,a) according to the above equation. The agent then updates the calculated cumulative Q-value and enters a new state s'. This process continues for the lifetime of the AU for the agent to continuously learn and adapt to the environment.

[0059] Algorithm 3 provides the steps that the agent follows in SARSA115, but does not provide the exact method that the agent uses to select the action a to be performed.

[0060] For this reason, the exemplary method devise a new policy for the agent to decide what action to take, which is shown below in Algorithm 4. Here, a constant called epsilon is defined, which has a value between 0 and 1 and can be tweaked by the operator. [Table 7]

[0061] First, the agent generates a random number between 0 and 1. If the random number is greater than the set value of epsilon, a random action is selected. Lower values ​​of epsilon trigger more random actions to be generated, while higher values ​​of epsilon trigger fewer random actions. Thus, in the early stages of deployment, when the agent explores various actions, the value of epsilon is usually set lower (e.g., 0.1). On the other hand, after the agent has explored sufficiently, the value of epsilon is set higher, e.g., 0.9, so that the agent starts to use and exploit the information learned in the early exploration stages. If the random number is less than the set value of epsilon, the reward r is checked. If the reward is less than 0, i.e., if the action resulted in a penalty, the action is immediately undone. However, if the reward is positive but less than the expected reward of the action, the same action is repeated. Thus, this is a greedy policy in which actions that are going in a positive direction are repeated, while actions that are going in a negative direction are immediately undone.

[0062] Regarding AU-specific quality evaluator design, the AU-independent SOTA AQuA filter provides a nominal spearman correlation (approximately -0.03) between predicted quality and different analyzed quality. This AU-independent AQuA filter can filter out low-quality frames for which the AU provides errors with high confidence, but cannot distinguish high-quality offloaded frames using fine-grained quality value prediction. Therefore, to obtain continuous quality measurements (e.g., reward function of RL algorithm) for any offloaded frames that pass the AQuA filter at runtime, the exemplary method designs AU-specific fine-grained proxy quality evaluation models. These quality evaluation models are defined and trained for various AUs as follows:

[0063] For the face recognition AU, two images are selected for 300 celebrities randomly sampled from the celebA dataset. One is selected as a reference image to build the database, and the other is selected as an unenrolled image and its multiple augmented variants (e.g., obtained by changing the values ​​of four knobs) to be used as query images. These query images (approximately 4M) constitute the training dataset for the quality assessment model. Each of these query images, when submitted to the face recognition model, provides the name of the celebrity with a match score between [0,1]. For each correct face recognition, the exemplary method calculates the corresponding rounded matchScore as the class label. * We use a value of 100. There can be a total of 101 classes in the training dataset. Therefore, the existing AQuA model can be trained with 101 output nodes.

[0064] For person and face detection AUs, to design quality evaluators specific to these detection AUs, the exemplary method trains an AQuA classification model with 201 output nodes of Olympic and HMDB training video frames and their extended variants (about 7.5M). During training, each frame is associated with a class label based on the joint metric of mAP and IoU for true positive detection, as shown in the following equation. Thus, the higher the class label, the higher the analysis quality.

number

[0065] Considering the effectiveness of CamTuner, one approach is to operate using a real camera and show the impact of the approach by comparing the AU accuracy in two cases. For example, in the first case, the exemplary technique is not applied and static camera parameters are applied. In the second case, the exemplary method applies the present technique to dynamically adjust the camera parameters. However, the problem with the present approach is that the camera can only have one set of parameters at any given time, and both conditions (with or without the exemplary technique) cannot be run simultaneously on one camera. Sets at different times have different environmental (sunny, cloudy, etc.) and scene conditions (faces visible in the field of view, etc.), making it an unfair comparison.

[0066] An alternative approach is that the exemplary method uses two cameras, AU is performed on one camera feed and the present technique is not applied, keeping the camera parameters the same and measuring the AU accuracy, and on the other camera feed, the same AU is performed and the exemplary technique is applied to dynamically adjust the camera parameters. Then, the accuracy measured in these two cases is compared. In this scenario, the environmental conditions seen by the two cameras are similar, but the scene conditions are not accurate even if they are similar. Some AUs are very sensitive and may not produce the same results with similar images. For example, the results depend on very fine characteristics in the frame and slight changes in the image may produce different results. Therefore, this is still not a fair comparison. Moreover, even if the AUs are tolerant to slight changes in the images, the changes in the environmental conditions are not deterministic, so it takes a very long time to perform various experiments, such as trying different configurations, training and testing the RL algorithm, trying different reward functions for RL, etc.

[0067] Due to the above issues of using real camera feeds, the exemplary method explores the effectiveness of the exemplary approach by using pre-recorded videos, such as the system 200 of FIG. 2. These videos are recorded with a single camera 210 setup in broad daylight conditions. By doing so, it is possible to obtain high-quality original video feeds with specific environmental conditions (sunlight in this case) and specific scene conditions (e.g., people's faces), and the exemplary method can also obtain ground truth for these videos. Then, two cameras, called virtual cameras, are simulated, where the same parameters as the real cameras are exposed and can be changed using an API. This allows experiments to be performed with different settings on the exact same scene, allowing repeatability and reproducibility in the experiments that is not possible with real cameras. Furthermore, the experiments can be performed faster than real time, i.e., frames can be executed at a desired high frame-per-second (FPS), allowing different reward functions used in RL algorithms, RL, etc. to be tested and trained separately.

[0068] These virtual cameras need to emulate the same day's environmental changes starting at sunrise, noon, afternoon, dusk, night, etc. The design of the virtual camera simulator includes two stages: an offline profiling stage and an online stage. In the offline profiling stage, first according to Algorithm 5 below, the exemplary method generates a VC (virtual-camera) table and a mapping function that maps a particular time of day to the corresponding brightness, contrast, saturation, and sharpness feature values ​​observed during that time. [Table 8]

[0069] Instead of calculating these feature values ​​for the entire frame because different parts of the frame are illuminated differently (e.g., due to the presence of obstacles, shadows, different color light sources, etc.), an exemplary method obtains feature values ​​for each of, for example, 12 tiles in the frame.

[0070] Each video captures the content and the environment at its capture time. Since the observed content needs to remain the same across the virtual cameras, only the environmental conditions throughout the day affect the captured frames. Using a camera parameter adjustment RL approach, the exemplary method strives to minimize the negative impact of the environment on the AU performance. To extend the videos to different times of the day while keeping the content the same, the exemplary method needs to generate another mapping table that maps the changes in brightness, contrast, saturation, and sharpness between the two times to the corresponding virtual camera knob configurations to obtain the changes. In an offline phase, the exemplary method generates this delta change-to-configuration mapping table according to Algorithm 6 reproduced below. [Table 9]

[0071] In the online phase, to simulate frames with different environmental conditions corresponding to times of the day other than the capture time, the exemplary method first divides the input frame into 12 tiles according to Algorithm 7 below. [Table 10]

[0072] For each tile, the exemplary method needs to extract four different image features, and based on the desired time (other than capture time), using the VC table (output of Algorithm 5), the exemplary method can obtain tile-specific desired values ​​of the image features. The small delta changes between the current feature values ​​and the desired feature values ​​are mapped to virtual camera knob configuration values ​​using the delta change-to-configuration mapping table (output of Algorithm 6). Although the exemplary method obtains different knob configurations for each tile, to eliminate tiling issues in the output frames (where separation of tiles is explicit), the exemplary method calculates a median configuration across all 12 configurations corresponding to each tile. With the modified median knob configuration, the exemplary method can obtain a video frame for the desired time.

[0073] In conclusion, CamTuner utilizes reinforcement learning techniques to continuously adapt to changing environmental conditions. CamTuner adjusts specific camera parameters so that the video feed generated by the camera is suitable for the analysis task at hand and its accuracy is improved. CamTuner is designed to be applicable to any video analysis task and to handle any complex set of camera parameters. To make the evaluation repeatable and reproducible (using the same scene to evaluate performance before and after applying CamTuner), the exemplary method uses a virtual camera and a loop-through of pre-recorded personal videos taken at different time segments of a day (Figure 2). With the help of VC, the exemplary method is able to run experiments faster in real time. That is, the exemplary method can run frames at a desired higher FPS and independently test and train different reward functions used in RL algorithms, RL, etc. The virtual camera (VC) is the same as a real RGB camera and depending on the time of capture of the day, it adapts the characteristics of the scene to that time without changing the frame content. VC continuously extracts frames from the input video and applies transformations to change the capture time to emulate different time segments of a day. Before offloading to the edge servers, the frames are passed through the AU-agnostic SOTA AQuA filter.

[0074] FIG. 3 is a block / flow diagram 800 of a practical application for automatically changing camera parameters to improve video analysis accuracy through continuously changing environmental conditions, in accordance with an embodiment of the present invention.

[0075] In one practical application, a video feed from a camera 802 is decoded by a "Video Sensor" 804 and frames are made available to a "Face Detection" component 806, which detects faces 808 (objects) and makes them available to a "Feature Extraction" component 810. Unique face templates such as features are then extracted and made available to a "Face Matching" component 812, which compares and matches these features against a gallery of face features. Changes in the environment can be detected by the CamTuner 950 using the AU-specific quality evaluator 110 and SARSA 115 by fine-tuning camera parameters such as brightness 820, contrast 822, saturation 824 and sharpness 826.

[0076] FIG. 4 is an exemplary processing system for automatically adjusting camera parameters to improve video analysis accuracy through continually changing environmental conditions, in accordance with an embodiment of the present invention.

[0077] The processing system includes at least one processor (CPU) 904 operatively connected to other components via a system bus 902. Also operatively connected to the system bus 902 are a GPU 905, a cache 906, a read only memory (ROM) 908, a random access memory (RAM) 910, an input / output (I / O) adapter 920, a network adapter 930, a user interface adapter 940, and / or a display adapter 950. Additionally, the CamTuner 950 includes an AU-specific quality evaluator 110 and a SARSA 115.

[0078] The storage devices 922 are operably connected to the system bus 902 by the I / O adapter 920. The storage devices 922 may be any type of disk storage device (e.g., magnetic disk storage device or optical disk storage device), solid state magnetic device, or the like.

[0079] The transceiver 932 is operably connected to the system bus 902 by the network adapter 930 .

[0080] User input device(s) 942 are operatively connected to system bus 902 by user interface adapter 940. User input device 942 may be any of a keyboard, mouse, keypad, image capture device, motion sensing device, microphone, or a device incorporating the functionality of at least two of these devices. Of course, other types of input devices may be used while maintaining the spirit of the principles of the present invention. Of course, other types of input devices may be used while maintaining the spirit of the present invention. User input device(s) 942 may be the same type of user input device or may be different types of user input devices. User input device 942 is used to input information to and output information from the processing system.

[0081] A display device 952 is operatively connected to the system bus 902 by a display adapter 950 .

[0082] Of course, the processing system may include other elements (not shown) or omit certain elements, as would be readily apparent to one of ordinary skill in the art. For example, the processing system may include various other types of input and / or output devices, depending on the particular implementation, as would be readily apparent to one of ordinary skill in the art. For example, various wireless and / or wired input and / or output devices may be used. Furthermore, various configurations of additional processors, controllers, memory, etc. may be used, as would be readily apparent to one of ordinary skill in the art. These and other variations of the processing system will be readily apparent to one of ordinary skill in the art from the teachings of the present principles provided herein.

[0083] FIG. 5 is a block / flow diagram of an exemplary method for automatically adjusting camera parameters to improve video analysis accuracy through continually changing environmental conditions, in accordance with an embodiment of the present invention.

[0084] In block 1001, video streams from multiple cameras are captured.

[0085] At block 1003, video analysis tasks, defined as analysis units (AUs), are performed on the video stream.

[0086] At block 1005, image processing is applied to the video stream to obtain processed frames.

[0087] At block 1007, the processed frames are filtered to discard low quality frames.

[0088] In block 1009, the filtered frames are passed to an AU-specific proxy quality evaluator, and the parameters of the multiple cameras are automatically fine-tuned using a State-Action-Reward-State-Action (SARSA) reinforcement learning computation, and based on the reinforcement computation, the parameters of the multiple cameras are dynamically fine-tuned by applying a new policy for the agent to take actions and learn to maximize reward.

[0089] As used herein, the terms "data," "content," "information," and similar terms may be used interchangeably to refer to data that may be obtained, transmitted, received, displayed, and / or stored by various exemplary embodiments. Thus, the use of these terms should not be construed as limiting the spirit and scope of the disclosure. Additionally, where a computing device is described herein for receiving data from another computing device, the data may be received directly from the other computing device or indirectly via one or more intermediate computing devices, such as one or more servers, relays, routers, network access points, base stations, and the like. Similarly, where a computing device is described herein for transmitting data to another computing device, the data may be transmitted directly to the other computing device or indirectly via one or more intermediate computing devices, such as one or more servers, relays, routers, network access points, base stations, and / or the like.

[0090] As will be appreciated by those skilled in the art, aspects of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may be generally referred to herein as a "circuit," "module," "computer," "apparatus," or "system." Additionally, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code thereon.

[0091] Any combination of one or more computer readable media may be used. The computer readable medium may be a computer readable signal medium or a computer readable recording medium. The computer readable recording medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of computer readable recording media, including but not limited to, include one or more wires, portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disc read-only memories (CD-ROMs), optical data storage devices, magnetic data storage devices, or any suitable combination of the foregoing. In the context of this document, a computer readable recording medium may be any tangible medium that contains or can store a program for use by or in connection with an instruction execution system, apparatus or device.

[0092] A computer-readable signal medium may include a propagated data signal in which computer-readable program code is embodied, for example in baseband or as part of a carrier wave. Such a propagated signal may be in any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium is not a computer-readable recording medium, but may be any computer-readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, device, or apparatus.

[0093] The program code embodied in the computer readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

[0094] Computer program code for carrying out processes related to aspects of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0095] Aspects of the present invention are described below with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions are provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to generate a machine such that the instructions, executed by a processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in one or more blocks or modules of the flowcharts and / or block diagrams.

[0096] These computer program instructions can be stored on a computer-readable medium that can instruct a computer, other programmable data processing device, or other device to function in a particular manner, such that the instructions stored on the computer-readable medium generate a product including instructions for implementing the functions / acts specified in one or more blocks or modules of the flowcharts and / or block diagrams.

[0097] The computer program instructions can also be loaded into a computer, other programmable data processing apparatus or other device to generate a computer-implemented process such that a series of operational steps are executed on the computer, other programmable apparatus or other device, and the instructions executing on the computer or other programmable apparatus provide a process for implementing the functions / operations specified in the blocks or modules of the flowcharts and / or block diagrams.

[0098] As used herein, the term "processor" is intended to include any processing device, such as one that includes a central processing unit (CPU) and / or other processing circuitry. It should also be understood that the term "processor" may refer to one or more processing devices, and that various elements associated with a processing device may be shared by other processing devices.

[0099] The term "memory" as used herein is intended to include memory associated with a processor or CPU, such as, for example, RAM, ROM, fixed memory devices (e.g., hard drives), removable memory devices (e.g., diskettes), flash memory, etc. Such memory may be considered a computer-readable recording medium.

[0100] Furthermore, the term "input / output device" or "I / O device" as used herein is intended to include, for example, one or more input devices (e.g., keyboard, mouse, scanner, etc.) for inputting data into a processing unit and / or one or more output devices (e.g., speakers, displays, printers, etc.) for presenting results associated with a processing unit.

[0101] The foregoing should be understood in all respects as illustrative and exemplary, and not restrictive, and the scope of the invention disclosed herein should be determined not from the detailed description, but from the claims which are to be interpreted in accordance with the broadest possible interpretation permitted by the Patent Law. It should be understood that the embodiments shown and described herein are merely illustrative of the principles of the invention, and that various modifications may be made by those skilled in the art without departing from the scope and spirit of the invention. Various other feature combinations may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Although aspects of the invention have been described above with the fine detail and particularity required by the Patent Law, the scope of the claims which are sought to be protected by Letters Patent are set forth in the appended claims.

Claims

1. 1. A method for automatically modifying camera parameters to improve video analysis accuracy through continuously changing environmental conditions, comprising: Capture video streams from multiple cameras (1001); performing 1003 video analysis tasks defined as analysis units (AUs) on said video stream; applying image processing to the video stream to obtain processed frames (1005); filtering (1007) the processed frames through a filter to discard low quality frames; and passing the filtered frames to an AU-specific proxy quality evaluator, and automatically fine-tuning the parameters of the multiple cameras using a State-Action-Reward-State-Action (SARSA) reinforcement learning (RL) computation, and dynamically fine-tuning the parameters of the multiple cameras by applying a new policy for an agent to take actions and learn to maximize rewards based on the reinforcement computation (1009).

2. A virtual camera (VC) Applying different environmental characteristics to the exact same scene captured from the video stream and evaluating it with different camera settings; 13. The method of claim 1, used to quickly and independently train and test RL algorithms and various reward functions used in RL.

3. The method of claim 2 , wherein the VC includes an offline profiling stage and an online stage.

4. 4. The method of claim 3, wherein a VC camera table and a mapping function are generated through the offline profiling step, the mapping function mapping a particular time of day to corresponding brightness, contrast, saturation and sharpness feature values ​​observed at that time.

5. The method of claim 3 , wherein frames of different environmental conditions corresponding to times other than the capture time are simulated during the online phase.

6. The method of claim 1 , wherein an analytical quality (AQ) metric specific to each of the AUs is used to train an AU-specific quality evaluator.

7. The method of claim 6 , wherein in the absence of ground truth in a real deployment, the AU-specific quality evaluator is used as a proxy to evaluate the AU.

8. 2. The method of claim 1, wherein the SARSA RL calculation includes a state, an action, and a reward, where the state is a vector including current brightness, contrast, sharpness, and color parameter values ​​of a camera among the multiple cameras, as well as measurements of brightness, contrast, sharpness, and color, the action is to increase or decrease one of the brightness, contrast, sharpness, or color parameter values, or to take no action at all, and the reward is the output of an AU-specific quality evaluator.

9. 1. A non-transitory computer readable recording medium including a computer readable program for automatically modifying camera parameters to improve video analysis accuracy through continuously changing environmental conditions, comprising: When the computer readable program is executed on the computer, the computer Capture video streams from multiple cameras (1001); Executing (1003) video analysis tasks defined as analysis units (AUs) on said video stream; applying image processing to the video stream to obtain processed frames (1005); filtering (1007) the processed frames through a filter to discard low quality frames; a state-action-reward-state-action (SARSA) reinforcement learning (RL) computation that dynamically fine-tunes parameters of the multiple cameras by applying a new policy for an agent to take actions and learn to maximize rewards based on the reinforcement computation (1009).

10. A virtual camera (VC) Applying different environmental characteristics to the exact same scene captured from the video stream and evaluating it with different camera settings; 10. The non-transitory computer-readable storage medium of claim 9, used to rapidly and individually train and test RL algorithms and various reward functions used in RL.

11. The non-transitory computer-readable medium of claim 10, wherein the VC includes an offline profiling stage and an online stage.

12. 12. The non-transitory computer-readable storage medium of claim 11, wherein a VC camera table and a mapping function are generated through the offline profiling step, the mapping function mapping a particular time of day to corresponding brightness, contrast, saturation and sharpness feature values ​​observed at that time.

13. The non-transitory computer-readable medium of claim 11 , wherein frames of different environmental conditions corresponding to times other than capture times are simulated during the online phase.

14. 10. The non-transitory computer-readable medium of claim 9, wherein an analytical quality (AQ) metric specific to each of the AUs is used to train an AU-specific quality evaluator.

15. The non-transitory computer-readable medium of claim 14 , wherein in the absence of ground truth in a real deployment, the AU-specific quality evaluator is used as a proxy to evaluate the AU.

16. 1. A system for automatically changing camera parameters to improve video analysis accuracy through continuous changes in environmental conditions, comprising: Memory, Capture video streams from multiple cameras (1001); performing 1003 video analysis tasks defined as analysis units (AUs) on said video stream; applying image processing to the video stream to obtain processed frames (1005); filtering (1007) the processed frames through a filter to discard low quality frames; one or more processors in communication with the memory configured to: pass the filtered frames to an AU-specific proxy quality evaluator; and dynamically fine-tune (1009) parameters of the multiple cameras using a State-Action-Reward-State-Action (SARSA) Reinforcement Learning (RL) computation; and dynamically fine-tune (1009) parameters of the multiple cameras by applying a new policy for an agent to take actions and learn to maximize rewards based on the reinforcement computation. A system having

17. A virtual camera (VC) Applying different environmental characteristics to the exact same scene captured from the video stream and evaluating it with different camera settings; 20. The system of claim 16, used to quickly and individually train and test RL algorithms and various reward functions used in RL.

18. 20. The system of claim 17, wherein the VC includes an offline profiling stage and an online stage.

19. 20. The system of claim 18, wherein a VC camera table and a mapping function are generated through the offline profiling step, the mapping function mapping a particular time of day to corresponding brightness, contrast, saturation and sharpness feature values ​​observed at that time.

20. The system of claim 18, wherein frames of different environmental conditions corresponding to times other than capture times are simulated during the online phase.

Images