Automatic exposure metering using artificial intelligence to track area of interest of a moving subject

Abstract

An apparatus includes an interface and a processor. The interface can be configured to receive pixel data. The processor can be configured to process the pixel data arranged as video frames, generate statistics and timestamps for the video frames, perform computer vision operations on the video frames to determine one or more regions of interest within the video frames at the timestamps, track movement of the regions of interest over a sequence of the video frames, evaluate a current luminance value of the regions of interest, compute an auto exposure parameter in response to the current luminance value and a target luminance value, and generate an output signal including the auto exposure parameter. The auto exposure parameter is computed for prioritizing exposure of the regions of interest in the video frames and dynamically changing the auto exposure parameter based on the movement of the regions of interest.

Description

Technical Field

[0001] The present invention relates generally to video capture, and more specifically to a method and / or apparatus for implementing automatic exposure metering for tracking regions of interest of moving subjects using artificial intelligence. Background Technology

[0002] Conventional automatic exposure (AE) metrology techniques use static metrology weighting tables. For example, conventional AE metrology techniques include spot metrology, center metrology, average metrology, and custom metrology (i.e., metrology customized by the end user). The static metrology weighting table used in conventional AE metrology is implemented using a predefined table selected from various metrology models. The selected model for the static metrology table is used for luminance calculation. The luminance value calculated based on the static metrology table represents the current luminance of the image. 3A techniques use the current luminance of the image to determine how AE will be applied.

[0003] In the static metering weighting table, each numerical value represents the weight of a sub-region (i.e., a tile) in the image. The brightness value of each sub-region is scaled according to the corresponding weight in the static metering weighting table. For example, a sub-region with a large weight value receives higher exposure priority than another sub-region with a small weight value. If the weight value is equal to zero, the corresponding sub-region will not be considered in the brightness calculation and will not affect the final AE adjustment.

[0004] Static metering weighted table designs are suitable for exposure in static scenes. However, if the scene includes a moving subject of interest, the static metering weighted table model has weaknesses. These weaknesses arise because the weights for each sub-region are fixed after initialization and cannot be dynamically changed (i.e., non-zero tiles). For example, in a video capture of a person moving from a low-weighted region to a high-weighted region, the person's exposure will be inconsistent (i.e., the person in the image might initially appear dark, then become normally exposed as they move to the high-weighted region). In another example, the background might be bright, and the sub-region of interest might be dark. Even if the sub-region of interest is located in an area with high-weighted tiles, the final average brightness will still be affected and averaged by the large brightness values ​​from the bright sub-region. Optimal exposure cannot be achieved using a static metering weighted table model.

[0005] The goal is to achieve automatic exposure metering by using artificial intelligence to track the region of interest of a moving subject. Summary of the Invention

[0006] This invention relates to an apparatus including an interface and a processor. The interface can be configured to receive pixel data. The processor can be configured to: process pixel data arranged as video frames; generate statistics and timestamps for the video frames; perform computer vision operations on the video frames to determine one or more regions of interest (ROIs) within the video frame at the timestamps; track the movement of the ROIs in a sequence of video frames; evaluate the current brightness value of the ROIs; calculate automatic exposure parameters in response to the current brightness value and a target brightness value; and generate an output signal including the automatic exposure parameters. The automatic exposure parameters can be calculated to prioritize the exposure of the ROIs in the video frames. The automatic exposure parameters can be calculated to dynamically change based on the movement of the ROIs. Attached Figure Description

[0007] Embodiments of the present invention will become apparent from the following detailed description and the appended claims and drawings.

[0008] Figure 1 This is a diagram illustrating an example of an edge device that can utilize a processor according to an exemplary embodiment of the invention, the processor being configured to implement automatic exposure metering for tracking regions of interest of a moving subject using artificial intelligence.

[0009] Figure 2 This is a diagram illustrating an example camera that implements an example embodiment of the present invention.

[0010] Figure 3 This is a block diagram showing the camera system.

[0011] Figure 4 This is a diagram illustrating the processing circuitry of a camera system that implements a convolutional neural network configured to perform object tracking and sorting.

[0012] Figure 5 This is a block diagram showing an intelligent automatic exposure metering system.

[0013] Figure 6 This is a diagram showing object detection and tracking in a sample video frame before applying AI exposure metering.

[0014] Figure 7 This is a diagram showing the region of interest tile classification for an example video frame.

[0015] Figure 8 This is a graph showing object detection and tracking in an example video frame after applying AI exposure metering.

[0016] Figure 9 This is a flowchart illustrating a method for implementing automatic exposure metering to track regions of interest of a moving subject using artificial intelligence.

[0017] Figure 10This is a flowchart illustrating a method for selecting an automatic exposure metering technique.

[0018] Figure 11 This is a flowchart illustrating a method for selecting automatic exposure parameters using full-field metering.

[0019] Figure 12 This is a flowchart illustrating a method for selecting automatic exposure parameters using previous brightness values ​​when there are no current computer vision results.

[0020] Figure 13 This is a flowchart illustrating a method for selecting automatic exposure parameters using region of interest tracking. Detailed Implementation

[0021] Embodiments of the present invention include providing automatic exposure metrology for tracking regions of interest (ROIs) of moving subjects using artificial intelligence, which can: (i) be interoperable with various computer vision techniques; (ii) track the exposure effects of multiple moving ROIs; (iii) provide accurate and consistent exposure for the ROIs; (iv) be applied to human detection, vehicle detection, motion detection, and / or face detection; (v) sort a list of detected ROIs; (vi) evaluate the current brightness value; (vii) calculate automatic exposure parameters to achieve a target brightness value; (viii) determine the brightness value based on computer vision results; (ix) prioritize exposure settings for moving objects; (x) classify tiles in captured images; (xi) switch between metrology techniques in response to computer vision results; (xii) predict the location of ROIs; and (xiii) be implemented as one or more integrated circuits.

[0022] Embodiments of the present invention can be configured to implement artificial intelligence (AI) based techniques to determine the parameters of automatic exposure (AE) for captured images. Computer vision operations can be implemented to use AI to track moving subjects in the image. The AI-based techniques used to track moving objects to determine the parameters of AE enable dynamic adjustments to the measurements applied to the image.

[0023] Embodiments of the present invention can be configured to interoperate with various computer vision techniques to track exposure effects on one or more regions of interest (ROIs) in a video frame. For example, AI metrology techniques can be implemented regardless of the specific type of computer vision implemented. For instance, computer vision can be implemented to analyze multiple ROIs in a video frame and provide ROI results for analysis using AI metrology techniques. Based on information about the multiple ROIs in the video frame, the AI ​​metrology techniques can be configured to perform accurate and consistent exposure on the ROIs.

[0024] The AI ​​metrology technology implemented by embodiments of the present invention can be applied to various scenarios to ensure adequate exposure of the Region of Interest (ROI). For example, implementing AI metrology technology can ensure good exposure of the ROI, enabling further post-processing techniques on video frames. AI metrology technology can use information about the detected ROI to ensure light insensitivity under various lighting conditions. AI metrology technology can be configured to provide accurate and consistent exposure of the ROI under various lighting conditions. Computer vision techniques can be implemented to track the ROI in a sequence of video frames to ensure accurate and consistent exposure is applied to the ROI even if the detected object corresponding to the ROI moves between dark and bright areas in the video frame.

[0025] Embodiments of the present invention can be configured to dynamically adjust how the After Effects (AE) are applied to video frames. In an example, when a moving object is detected in a video frame, the AE parameters can be selected to ensure consistent exposure for the Region of Interest (ROI). However, when no ROI is detected in a video frame, the AI ​​metrology technique can be configured to recalculate the AE parameters to apply exposure to the entire video frame (e.g., providing exposure for the entire image without preference). For example, the AI ​​metrology technique might be applicable to video frames capturing moving objects and / or video frames capturing static scenes.

[0026] Embodiments of the present invention can be implemented in a processor implemented by a camera system (e.g., an edge device). In one example, a battery-powered camera can implement the processor. The battery-powered camera can be implemented on the front door of a home that can capture a static scene (e.g., doorsteps and street). Potential moving subjects in the scene can include visitors approaching the home. For home use scenarios, users may prefer consistent exposure for visitors rather than well-exposed street scenes. AI metering techniques can be configured to provide consistent and accurate exposure for visitors.

[0027] In some embodiments, AI metrology techniques can be implemented by a processor used in a hunting camera. The hunting camera can be mounted in a forest to continuously search for nearby animals. The static scene can be a forest, and the potential moving subjects in the scene can be animals. For a hunting camera scene, a user might prefer consistent exposure of the animals rather than well-exposed forest. The AI ​​metrology techniques can be configured to provide consistent and accurate exposure for the animals.

[0028] In some embodiments, AI metrology techniques can be implemented by a processor used in an access control camera. The access control camera can be placed next to a security door to detect authorized individuals and grant them access. The static scene can be a doorway, and potential moving subjects in the scene can be people walking through the door. For access control camera scenarios, users may prefer well-exposed faces to well-exposed doorways. AI metrology techniques can be configured to provide consistent and accurate exposure for faces.

[0029] In some embodiments, AI metrology technology can be implemented by a processor for a traffic violation detection camera. The traffic detection camera can be mounted on an outdoor pole of a traffic light to detect traffic violations by vehicles. The static scene can be a road, street, sign, tree, and building, and the potential moving subject in the scene can be a vehicle. For traffic detection scenarios, users may prefer well-exposed vehicles rather than well-exposed roads, streets, signs, trees, and buildings. The AI ​​metrology technology can be configured to provide consistent and optimal exposure for vehicles.

[0030] In some embodiments, AI metrology techniques can be implemented by a processor for a falling object detection camera. The falling object detection camera can be installed on the ground floor of an apartment and / or other high-rise building to detect falling objects. The static scene can be a building, and potential moving subjects in the scene can be falling objects. For falling object scenes, users may prefer well-exposed falling objects to well-exposed buildings. The AI ​​metrology techniques can be configured to provide consistent and optimal exposure for falling objects.

[0031] In some embodiments, AI metrology techniques can be implemented by a processor used in a security camera. The security camera can be mounted on a pole or other high-positioned location in a community to detect suspicious vehicles or people. The static scene can be buildings and streets in the community, and potential moving subjects in the scene can be pedestrians and vehicles. For a security camera scene, users may prefer well-exposed pedestrians and vehicles rather than well-exposed buildings and streets. The AI ​​metrology techniques can be configured to provide consistent and optimal exposure for pedestrians and vehicles.

[0032] Embodiments of the present invention can be configured to operate in various detection scenarios. In one example, AI measurement technology can be implemented by a camera that is oriented towards a fixed location capable of capturing moving subjects (e.g., a doorbell camera for visitors). In another example, AI measurement technology can be implemented by a camera that is mobile and oriented towards a fixed subject (e.g., a vehicle-mounted camera that moves with the vehicle and captures a fixed landscape). In yet another example, AI measurement technology can be implemented by a camera that is mobile and oriented towards a moving subject (e.g., a vehicle-mounted camera that moves with the vehicle and captures pedestrians and other moving vehicles). Typically, a camera that moves while capturing moving subjects is likely to be the most challenging implementation. Embodiments of the present invention can overcome the inherent difficulties of mobile cameras capturing moving subjects by using ROI prediction based on consistent relative movement between the camera and the subject (e.g., relative movement between a vehicle and a pedestrian).

[0033] Embodiments of the present invention can be configured to work with image sensors (e.g., RGB image sensors and / or RGB-IR sensors). In some embodiments, AI metrology techniques can be implemented using a camera system that implements a structured light projector (SLP). However, AI metrology techniques can be implemented in cameras that do not implement an SLP. The type of camera implemented and / or the use case for video capture can vary depending on the design criteria of the specific implementation.

[0034] refer to Figure 1 The diagram illustrates an example of an edge device that can utilize a processor according to an exemplary embodiment of the invention, the processor being configured to implement automatic exposure metering for tracking a region of interest of a moving subject using artificial intelligence. A top view of zone 50 is shown. In the illustrated example, zone 50 may be an outdoor location. Streets, vehicles, and buildings are shown.

[0035] Devices 100a-100n are shown at various locations within zone 50. Devices 100a-100n can each implement an edge device. Edge devices 100a-100n may include intelligent IP cameras (e.g., camera systems). Edge devices 100a-100n may include low-power technologies designed for deployment in embedded platforms at the network edge (e.g., microprocessors running on sensors, cameras, or other battery-powered devices), where power consumption is a critical issue. In the examples, edge devices 100a-100n may include various traffic cameras and Intelligent Transportation System (ITS) solutions.

[0036] Edge devices 100a-100n can be implemented for a variety of applications. In the illustrated example, edge devices 100a-100n may include an Automatic License Plate Recognition (ANPR) camera 100a, a traffic camera 100b, a vehicle camera 100c, an access control camera 100d, an ATM camera 100e, a bullet camera 100f, a dome camera 100n, etc. In the example, edge devices 100a-100n can be implemented as a traffic camera and Intelligent Transportation System (ITS) solution designed to enhance road safety by utilizing a combination of people and vehicle detection, vehicle brand / model recognition, and Automatic License Plate Recognition (ANPR) functions.

[0037] In the illustrated example, zone 50 can be an outdoor location. In some embodiments, edge devices 100a-100n can be implemented at various indoor locations. In the example, edge devices 100a-100n can incorporate convolutional neural networks for use in security (surveillance) applications and / or access control applications. In the example, edge devices 100a-100n implemented as security cameras and access control applications can include battery-powered cameras, doorbell cameras, outdoor cameras, indoor cameras, etc. According to embodiments of the invention, security cameras and access control applications can realize performance benefits from the application of convolutional neural networks. In the example, edge devices utilizing convolutional neural networks according to embodiments of the invention can acquire large amounts of image data and perform on-device inference to obtain useful information (e.g., multiple temporal instances of images performed by each network), thereby reducing bandwidth and / or power consumption. The design, type, and / or application performed by edge devices 100a-100n can vary according to the design criteria of a particular implementation.

[0038] refer to Figure 2 The diagram illustrates an example edge device camera that implements an exemplary embodiment of the present invention. Camera systems 100a-100n are shown. Each camera device 100a-100n can have a different style and / or use case. For example, camera 100a can be an action camera, camera 100b can be a ceiling-mounted security camera, camera 100n can be a network camera, etc. Other types of cameras can be implemented (e.g., home security cameras, battery-powered cameras, doorbell cameras, stereo cameras, etc.). The design / style of cameras 100a-100n can vary according to the design criteria of a particular implementation.

[0039] Each of the camera systems 100a-100n may include block (or circuit) 102 and / or block (or circuit) 104. Some of the camera systems 100a-100n may also include block (or circuit) 106. Circuit 102 may implement a processor. Circuit 104 may implement a capture device. Circuit 106 may implement a structured light projector. The camera systems 100a-100n may include other components (not shown). They can be used with... Figure 3 The components of the cameras 100a-100n are described in detail.

[0040] Processor 102 can be configured to implement an artificial neural network (ANN). In an example, the ANN may include a convolutional neural network (CNN). Processor 102 can be configured to implement a video encoder. Processor 102 can be configured to process pixel data arranged as video frames. Capture device 104 can be configured to capture pixel data that can be used by processor 102 to generate video frames. Structured light projector 106 can be configured to generate a structured light pattern (e.g., a speckle pattern). The structured light pattern can be projected onto a background (e.g., the environment). Capture device 104 can capture pixel data including a background image (e.g., the environment) with a speckle pattern.

[0041] Cameras 100a-100n can be edge devices. A processor 102 implemented by each of the cameras 100a-100n enables the cameras 100a-100n to perform various functions internally (e.g., at the local level). For example, processor 102 can be configured to perform object / event detection (e.g., computer vision operations), 3D reconstruction, presence detection, depth map generation, video encoding, and / or video transcoding on the device. For example, processor 102 can even perform advanced processes such as computer vision and 3D reconstruction without uploading video data to a cloud service, thereby offloading computationally intensive functions (e.g., computer vision, video encoding, video transcoding, etc.).

[0042] In some embodiments, multiple camera systems may be implemented (e.g., camera systems 100a-100n may operate independently of each other). For example, each camera in 100a-100n may individually analyze the captured pixel data and perform event / object detection locally. In some embodiments, cameras 100a-100n may be configured as a camera network (e.g., security cameras that send video data to a central source such as a network-attached storage device and / or cloud service). The location and / or configuration of cameras 100a-100n may vary according to the design criteria of a particular implementation.

[0043] The capture device 104 of each camera in the camera systems 100a-100n may include a single lens (e.g., a monocular camera). The processor 102 may be configured to accelerate the preprocessing of speckle structured light for monocular 3D reconstruction. Monocular 3D reconstruction can be performed to generate depth maps and / or parallax images without using a stereo camera.

[0044] In some embodiments, camera systems 100a-100n may implement multiple capture devices to provide stereo vision operation (e.g., depth processing using multiple similar images). In the illustrated example, camera systems 100a and 100n are shown with a structured light projector 106, and camera systems 10b and 100c are shown without a structured light projector 106. The AI ​​metrology technology implemented by the processor 102 of each of the camera systems 100a-100n can be configured to provide functionality with a monocular camera, a stereo vision camera, RGB images (without a structured light projector 106), RGB-IR images (with a structured light projector 106), etc. Typically, camera systems 100a-100n and / or processor 102 may be described according to an embodiment that implements a monocular capture device 104 without a structured light projector 106. For embodiments of the structured light projector 106, details of the structured light projector 106 may be designed based on details provided in the following applications: U.S. Patent Application 16 / 520,754, filed July 24, 2019; U.S. Patent Application 16 / 996,114, filed August 18, 2020; U.S. Patent Application 17 / 236,340, filed April 21, 2021; and U.S. Patent Application 17 / 694,726, filed March 15, 2022, appropriate portions of which are incorporated herein by reference. However, the processor 102 may be configured to provide AI metrology techniques for various types of image sensors and / or cameras according to design criteria for a particular implementation.

[0045] refer to Figure 3 The diagram illustrates a block diagram of a camera system 100, demonstrating an example implementation. The camera system 100 can be... Figure 2 Representative examples of cameras 100a-100n are shown in association. Camera system 100 may include processor / SoC 102 and capture device 104.

[0046] The camera system 100 may further include blocks (or circuits) 150, 152, 154, 156, 158, 160, 164, and / or 166. Circuit 150 may implement a memory. Circuit 152 may implement a battery. Circuit 154 may implement a communication device. Circuit 156 may implement a wireless interface. Circuit 158 ​​may implement a general-purpose processor. Block 160 may implement an optical lens. Circuit 164 may implement one or more sensors. Circuit 166 may implement a human-machine interface (HID) device. In some embodiments, the camera system 100 may include a processor / SoC 102, a capture device 104, a memory 150, a lens 160, a sensor 164, a battery 152, a communication module 154, a wireless interface 156, and a processor 158. In another example, camera system 100 may include a processor / SoC 102, a capture device 104, a processor 158, a lens 160, and a sensor 164 as a single device, while memory 150, battery 152, communication module 154, and wireless interface 156 may be components of separate devices. Camera system 100 may include other components (not shown). The number, type, and / or arrangement of components in camera system 100 may vary depending on the design criteria of a particular implementation.

[0047] Processor 102 can be implemented as a video processor. In an example, processor 102 can be configured to receive three-sensor video input using a high-speed SLVS / MIPI-CSI / LVCMOS interface. In some embodiments, processor 102 can be configured to perform depth sensing in addition to generating video frames. In an example, depth sensing can be performed in response to depth information and / or vector light data captured in the video frames.

[0048] Memory 150 can store data. Memory 150 can be implemented as various types of memory, including but not limited to cache, flash memory, memory card, random access memory (RAM), dynamic RAM (DRAM), etc. The type and / or size of memory 150 can vary depending on the design criteria of a particular implementation. The data stored in memory 150 may correspond to video files, motion information (e.g., readings from sensor 164), video fusion parameters, image stabilization parameters, user input, computer vision models, feature sets, and / or metadata information. In some embodiments, memory 150 can store reference images. Reference images can be used for computer vision operations, 3D reconstruction, automatic exposure, etc. In some embodiments, reference images may include reference structured light images.

[0049] Processor / SoC 102 can be configured to execute computer-readable code and / or procedural information. In various embodiments, the computer-readable code can be stored within processor / SoC 102 (e.g., microcode, etc.) and / or in memory 150. In an example, processor / SoC 102 can be configured to execute one or more artificial neural network models (e.g., face recognition CNN, object detection CNN, object classification CNN, 3D reconstruction CNN, presence detection CNN, etc.) stored in memory 150. In an example, memory 150 can store one or more directed acyclic graphs (DAGs) and one or more sets of weights or biases defining one or more artificial neural network models. Processor / SoC 102 can be configured to receive input from memory 150 and / or present output to memory 150. Processor / SoC 102 can be configured to present and / or receive other signals (not shown). The number and / or type of inputs and / or outputs of processor / SoC 102 can vary depending on the design criteria of a particular implementation. Processor / SoC 102 can be configured for low-power (e.g., battery) operation.

[0050] Battery 152 can be configured to store and / or supply power to components of camera system 100. The dynamic actuator mechanism for the rolling shutter sensor 130 can be configured to conserve power. Reduced power consumption allows camera system 100 to operate for extended periods using battery 152 without recharging. Battery 152 can be rechargeable. Battery 152 can be built-in (e.g., non-replaceable) or replaceable. Battery 152 can have an input for connection to an external power source (e.g., for charging). In some embodiments, device 100 can be powered by an external power source (e.g., battery 152 may not be implemented or may be implemented as a backup power source). Battery 152 can be implemented using various battery technologies and / or chemistry. The type of battery 152 implemented can vary depending on the design criteria of a particular implementation.

[0051] The communication module 154 can be configured to implement one or more communication protocols. For example, the communication module 154 and the wireless interface 156 can be configured to implement one or more of the following: IEEE 102.11, IEEE 102.15, IEEE 102.15.1, IEEE 102.15.2, IEEE 102.15.3, IEEE 102.15.4, IEEE 102.15.5, IEEE 102.20, and / or In some embodiments, the communication module 154 may be a hardwired data port (e.g., a USB port, a mini-USB port, a USB-C connector, an HDMI port, an Ethernet port, a DisplayPort interface, a Lightning port, etc.). In some embodiments, the wireless interface 156 may also implement one or more protocols associated with a cellular communication network (e.g., GSM, CDMA, GPRS, UMTS, CDMA2000, 3GPP LTE, 4G / HSPA / WiMAX, SMS, etc.). In embodiments where the camera system 100 is implemented as a wireless camera, the protocols implemented by the communication module 154 and the wireless interface 156 may be wireless communication protocols. The type of communication protocol implemented by the communication module 154 may vary depending on the design criteria of the specific implementation.

[0052] Communication module 154 and / or wireless interface 156 can be configured to generate broadcast signals as output from camera system 100. The broadcast signals can send video data, parallax data, and / or (multiple) control signals to external devices. For example, the broadcast signals can be sent to cloud storage services (e.g., storage services capable of scaling on demand). In some embodiments, communication module 154 may not send data until the processor / SoC 102 has performed video analysis to determine that an object is in the field of view of camera system 100.

[0053] In some embodiments, the communication module 154 can be configured to generate a manual control signal. The manual control signal can be generated in response to a signal received from a user by the communication module 154. The manual control signal can be configured to activate the processor / SoC 102. The processor / SoC 102 can be activated in response to the manual control signal, regardless of the power state of the camera system 100.

[0054] In some embodiments, the communication module 154 and / or the wireless interface 156 may be configured to receive a feature set. The received feature set may be used to detect events and / or objects. For example, the feature set may be used to perform computer vision operations. The feature set information may include instructions for the processor 102 to determine which types of objects correspond to objects of interest and / or events.

[0055] In some embodiments, the communication module 154 and / or the wireless interface 156 may be configured to receive user input. User input allows the user to adjust operating parameters for various features implemented by the processor 102. In some embodiments, the communication module 154 and / or the wireless interface 156 may be configured to interact with an application (e.g., an app) (e.g., using an application programming interface (API)). For example, the application may be implemented on a smartphone to allow the end user to adjust various settings and / or parameters for various features implemented by the processor 102 (e.g., setting video resolution, selecting frame rate, selecting output format, setting tolerance parameters for 3D reconstruction, etc.).

[0056] Processor 158 can be implemented using general-purpose processor circuitry. Processor 158 may be operable to interact with video processing circuitry 102 and memory 150 to perform various processing tasks. Processor 158 may be configured to execute computer-readable instructions. In one example, computer-readable instructions may be stored in memory 150. In some embodiments, computer-readable instructions may include controller operations. Typically, input from sensor 164 and / or human-machine interface device 166 is shown to be received by processor 102. In some embodiments, general-purpose processor 158 may be configured to receive and / or analyze data from sensor 164 and / or HID 166 and make decisions in response to input. In some embodiments, processor 158 may send data to and / or receive data from other components of camera system 100, such as battery 152, communication module 154, and / or wireless interface 156. Which functions of camera system 100 are performed by processor 102 and general-purpose processor 158 may vary depending on the design criteria of a particular implementation.

[0057] Lens 160 may be attached to capture device 104. Capture device 104 may be configured to receive an input signal (e.g., LIN) via lens 160. The signal LIN may be an optical input (e.g., an analog image). Lens 160 may be implemented as an optical lens. Lens 160 may provide zoom and / or focus features. In one example, capture device 104 and / or lens 160 may be implemented as a single lens assembly. In another example, lens 160 may be implemented separately from capture device 104.

[0058] Capture device 104 can be configured to convert input light LIN into computer-readable data. Capture device 104 can capture data received through lens 160 to generate raw pixel data. In some embodiments, capture device 104 can capture data received through lens 160 to generate a bitstream (e.g., generate video frames). For example, capture device 104 can receive focused light from lens 160. Lens 160 can be oriented, tilted, panned, scaled, and / or rotated to provide a target view from camera system 100 (e.g., a view of video frames, a view of panoramic video frames captured using multiple camera systems 100a-100n, a target image and reference image view for stereo vision, etc.). Capture device 104 can generate a signal (e.g., VIDEO). The signal VIDEO can be pixel data (e.g., a sequence of pixels that can be used to generate video frames). In some embodiments, the signal VIDEO can be video data (e.g., a sequence of video frames). The signal VIDEO can be presented to one of the inputs of processor 102. In some embodiments, the pixel data generated by the capture device 104 may be uncompressed and / or raw data generated in response to focused light from the lens 160. In some embodiments, the output of the capture device 104 may be a digital video signal.

[0059] In this example, capture device 104 may include block (or circuitry) 180, block (or circuitry) 182, and block (or circuitry) 184. Circuitry 180 may be an image sensor. Circuitry 182 may be a processor and / or logic unit. Circuitry 184 may be memory circuitry (e.g., a frame buffer). Lens 160 (e.g., a camera lens) may be oriented to provide a view of the environment surrounding camera system 100. Lens 160 may be designed to capture environmental data (e.g., light input LIN). Lens 160 may be a wide-angle lens and / or a fisheye lens (e.g., a lens capable of capturing a wide field of view). Lens 160 may be configured to capture and / or focus light for capture device 104. Typically, image sensor 180 is located behind lens 160. Based on the light captured from lens 160, capture device 104 may generate bitstream and / or video data (e.g., a signal VIDEO).

[0060] Capture device 104 can be configured to capture video image data (e.g., light collected and focused by lens 160). Capture device 104 can capture data received through lens 160 to generate a video bitstream (e.g., pixel data for a sequence of video frames). In various embodiments, lens 160 can be implemented as a fixed-focus lens. Fixed-focus lenses are generally advantageous for smaller size and lower power consumption. In examples, fixed-focus lenses can be used in battery-powered, doorbell, and other low-power camera applications. In some embodiments, lens 160 can be oriented, tilted, panned, zoomed, and / or rotated to capture the environment around camera system 100 (e.g., capturing data from the field of view). In examples, professional camera models can utilize active lens systems for enhanced functionality, remote control, etc.

[0061] The capture device 104 can convert received light into a digital data stream. In some embodiments, the capture device 104 can perform analog-to-digital conversion. For example, the image sensor 180 can perform photoelectric conversion on the light received by the lens 160. The processor / logic unit 182 can convert the digital data stream into a video data stream (or bitstream), a video file, and / or multiple video frames. In this example, the capture device 104 can present the video data as a digital video signal (e.g., VIDEO). The digital video signal can include video frames (e.g., continuous digital images and / or audio). In some embodiments, the capture device 104 can include a microphone for capturing audio. In some embodiments, the microphone can be implemented as a separate component (e.g., one of the sensors 164).

[0062] Video data captured by capture device 104 can be represented as a signal / bitstream / data VIDEO (e.g., a digital video signal). Capture device 104 can present the signal VIDEO to processor / SoC 102. The signal VIDEO can represent video frames / video data. The signal VIDEO can be a video stream captured by capture device 104. In some embodiments, the signal VIDEO may include pixel data operable by processor 102 (e.g., a video processing pipeline, image signal processor (ISP), etc.). Processor 102 can generate video frames in response to the pixel data in the signal VIDEO.

[0063] The signal VIDEO may include pixel data arranged into video frames. The signal VIDEO may be an image including the background (e.g., the captured environment) and various objects. The signal VIDEO may include a single-channel source image. A single-channel source image may be generated in response to capturing pixel data using a monocular lens 160.

[0064] Image sensor 180 can receive input light LIN from lens 160 and convert the light LIN into digital data (e.g., a bitstream). For example, image sensor 180 can perform photoelectric conversion on light from lens 160. In some embodiments, image sensor 180 may have an additional margin that is not used as part of the image output. In some embodiments, image sensor 180 may not have an additional margin. In various embodiments, image sensor 180 can be configured to generate RGB video signals, IR video signals, and / or RGB-IR video signals. In a field of view illuminated only by infrared light, image sensor 180 can generate a monochrome (B / W) video signal. In a field of view illuminated by both IR and visible light, image sensor 180 can be configured to generate color information in addition to generating a monochrome video signal. In various embodiments, image sensor 180 can be configured to generate a video signal in response to visible light and / or infrared (IR) light.

[0065] In some embodiments, camera sensor 180 may include a rolling shutter sensor or a global shutter sensor. In an example, the rolling shutter sensor 180 may implement an RGB-IR sensor. In some embodiments, capture device 104 may include a rolling shutter IR sensor and an RGB sensor (e.g., implemented as separate components). In an example, the rolling shutter sensor 180 may be implemented as an RGB-IR rolling shutter complementary metal-oxide-semiconductor (CMOS) image sensor. In one example, the rolling shutter sensor 180 may be configured to assert a signal indicating the exposure time of the first row. In one example, the rolling shutter sensor 180 may apply a mask to a monochrome sensor. In an example, the mask may include multiple cells containing a red pixel, a green pixel, a blue pixel, and an IR pixel. The IR pixel may contain red, green, and blue filter materials that efficiently absorb all light in the visible spectrum while allowing longer infrared wavelengths to pass through with minimal loss. In the case of a rolling shutter, as each row (or line) of the sensor begins exposure, all pixels in that row (or line) may begin exposure simultaneously.

[0066] Processor / logic unit 182 can convert the bitstream into human-readable content (e.g., video data that an average person can understand regardless of image quality, such as video frames and / or pixel data that can be converted into video frames by processor 102). For example, processor / logic unit 182 can receive raw (e.g., raw) data from image sensor 180 and generate (e.g., encode) video data (e.g., bitstream) based on the raw data. Capture device 104 may have memory 184 to store raw data and / or processed bitstreams. For example, capture device 104 may implement frame memory and / or buffer 184 to store (e.g., provide temporary storage and / or cache) one or more video frames (e.g., digital video signals). In some embodiments, processor / logic unit 182 can perform analysis and / or correction on the video frames stored in memory / buffer 184 of capture device 104. Processor / logic unit 182 can provide status information about the captured video frames.

[0067] Capture device 104 can be configured to receive a signal (e.g., AE_PARAM). The AE_PARAM signal can be generated by processor 102. The AE_PARAM signal can include automatic exposure parameters generated by AI metrology techniques implemented by processor 102. Processor / logic unit 182 can be configured to adjust operating parameters of capture device 104 in response to the AE_PARAM signal. In one example, processor / logic unit 182 can adjust the exposure of image sensor 180 in response to the AE_PARAM signal. In another example, processor / logic unit 182 can adjust the DC aperture and / or shutter speed for image sensor 180 in response to the AE_PARAM signal. In yet another example, processor / logic unit 182 can adjust the zoom / tilt / pan / focus of capture device 104 in response to the AE_PARAM signal. In yet another example, processor / logic unit 182 can adjust the automatic gain control (AGC) of image sensor 180 in response to the AE_PARAM signal. In some embodiments, the AE_PARAM signal can be configured to provide activation and / or wake-up signals to capture device 104. The type of adjustment performed in response to the signal AE_PARAM to adjust the exposure, focus, and / or capture direction of the capture device 104 can vary according to the design criteria of a particular implementation.

[0068] Sensor 164 can implement multiple sensors, including but not limited to motion sensors, ambient light sensors, proximity sensors (e.g., ultrasonic, radar, lidar, etc.), audio sensors (e.g., microphones), etc. In embodiments implementing a motion sensor, sensor 164 can be configured to detect motion anywhere (or some location outside the field of view) monitored by camera system 100. In various embodiments, motion detection can be used as a threshold for activating capture device 104. Sensor 164 can be implemented as an internal component of camera system 100 and / or an external component of camera system 100. In one example, sensor 164 can be implemented as a passive infrared (PIR) sensor. In another example, sensor 164 can be implemented as a smart motion sensor. In yet another example, sensor 164 can be implemented as a microphone. In embodiments implementing a smart motion sensor, sensor 164 may include a low-resolution image sensor configured to detect motion and / or people.

[0069] In various embodiments, sensor 164 may generate signals (e.g., SENS). The SENS may include various data (or information) collected by sensor 164. In an example, the SENS may include data collected in response to motion detected in a monitored field of view, ambient light levels in the monitored field of view, and / or sound picked up in the monitored field of view. However, other types of data may be collected and / or generated based on application-specific design criteria. The SENS may be presented to processor / SoC 102. In an example, sensor 164 may generate (assert) the SENS when motion is detected in a field of view monitored by camera system 100. In another example, sensor 164 may generate (assert) the SENS when audio is triggered in a field of view monitored by camera system 100. In yet another example, sensor 164 may be configured to provide directional information about motion and / or sound detected in the field of view. This directional information may also be communicated to processor / SoC 102 via the SENS.

[0070] HID 166 can implement an input device. For example, HID 166 can be configured to receive human input. HID 166 can generate a signal (e.g., USER). The USER signal can provide the input received by HID 166 to processor 102. The USER signal can include human input. In one example, HID 166 can be configured to receive password input from a user. In another example, HID 166 can be configured to receive user input to provide various parameters and / or settings to processor 102 and / or memory 150. In some embodiments, camera system 100 can include a keyboard, touchpad (or screen), doorbell switch, and / or other human-machine interface device (HID) 166. In an example, sensor 164 can be configured to determine when an object approaches HID 166. In an example where camera system 100 is implemented as part of an access control application, capture device 104 can be activated to provide an image for identifying a person attempting to access, and the lock zone and / or illumination for access touchpad 166 can be turned on. For example, a combination of inputs from HID 166 (e.g., a password or PIN) can be combined with an existence check and / or deep analysis performed by processor 102 to enable two-factor authentication.

[0071] The processor / SoC 102 may receive a signal VIDEO, a signal SENS, and / or a signal USER. The processor / SoC 102 may generate one or more video output signals (e.g., VIDOUT) and / or one or more control signals (e.g., CTRL) based on the signal VIDEO, signal SENS, signal USER, and / or other inputs. In some embodiments, the signals VIDOUT and CTRL may be generated based on analysis of the signal VIDEO and / or objects detected in the signal VIDEO.

[0072] In various embodiments, the processor / SoC 102 may be configured to perform one or more of the following: feature extraction, object detection, object tracking, 3D reconstruction, presence detection, and object identification. For example, the processor / SoC 102 may determine motion information and / or depth information by analyzing frames from a signal VIDEO and comparing those frames with previous frames. The comparison may be used to perform digital motion estimation. In some embodiments, the processor / SoC 102 may be configured to generate a video output signal VIDOUT including video data and / or a depth data signal including a disparity map and a depth map based on the signal VIDEO. The video output signal VIDOUT and / or the depth data may be presented to memory 150, communication module 154, and / or wireless interface 156. In some embodiments, the video signal VIDOUT and / or the depth data may be used internally by the processor 102 (e.g., not presented as output).

[0073] The signal VIDOUT can be presented to the communication device 156. In some embodiments, the signal VIDOUT may include encoded video frames generated by the processor 102. In some embodiments, the encoded video frames may include a complete video stream (e.g., encoded video frames representing all video captured by the capture device 104). The encoded video frames may be encoded, cropped, stitched, and / or enhanced versions of pixel data received from the signal VIDEO. In an example, the encoded video frames may be high-resolution, digital, encoded, de-distorted, stabilized, cropped, blended, stitched, and / or rolling shutter effect corrected versions of the signal VIDEO.

[0074] In some embodiments, the signal VIDOUT may be generated based on video analysis (e.g., computer vision operations) performed by processor 102 on generated video frames. Processor 102 may be configured to perform computer vision operations to detect objects and / or events in the video frames, and then convert the detected objects and / or events into statistics and / or parameters. In one example, the data determined by the computer vision operations may be converted by processor 102 into a human-readable format. Data from the computer vision operations can be used to detect objects and / or events. The computer vision operations may be performed locally by processor 102 (e.g., without needing to communicate with external devices to offload computational operations). For example, locally executed computer vision operations allow the computer vision operations to be performed by processor 102 and avoid heavy video processing running on a backend server. Avoiding video processing on a backend (e.g., remotely located) server can protect privacy.

[0075] In some embodiments, the signal VIDOUT can be data generated by processor 102 (e.g., video analysis results, audio / speech analysis results, etc.) that can be communicated to cloud computing services to aggregate information and / or provide training data for machine learning (e.g., to improve object detection, improve audio detection, improve presence detection, etc.). In some embodiments, the signal VIDOUT can be provided to cloud services for mass storage (e.g., to enable users to retrieve encoded video using smartphones and / or desktop computers). In some embodiments, the signal VIDOUT can include data extracted from video frames (e.g., computer vision results) and can communicate the results to another device (e.g., a remote server, cloud computing system, etc.) to offload the analysis of the results to another device (e.g., offloading the analysis of the results to a cloud computing service, rather than performing all the analysis locally). The type of information communicated by the signal VIDOUT can vary depending on the design criteria of a particular implementation.

[0076] The CTRL signal can be configured to provide a control signal. The CTRL signal can be generated in response to a decision made by the processor 102. In one example, the CTRL signal can be generated in response to a detected object and / or features extracted from a video frame. The CTRL signal can be configured to enable, disable, or change the operating mode of another device. In one example, the CTRL signal can be used to lock / unlock a door controlled by an electronic lock. In another example, the CTRL signal can be used to put a device into sleep mode (e.g., low power mode) and / or activate it from sleep mode. In yet another example, the CTRL signal can be used to generate an alarm and / or notification. The type of device controlled by the CTRL signal and / or the response performed by the device in response to the CTRL signal can vary depending on the design criteria of the specific implementation.

[0077] A signal CTRL can be generated based on data received by sensor 164 (e.g., temperature readings, motion sensor readings, etc.). A signal CTRL can be generated based on input from HID 166. A signal CTRL can be generated based on human behavior detected by processor 102 in a video frame. A signal CTRL can be generated based on the type of object detected (e.g., person, animal, vehicle, etc.). A signal CTRL can be generated in response to detecting a specific type of object at a specific location. A signal CTRL can be generated in response to user input to provide various parameters and / or settings to processor 102 and / or memory 150. Processor 102 can be configured to generate a signal CTRL in response to sensor fusion operations (e.g., aggregating information received from different sources). Processor 102 can be configured to generate a signal CTRL in response to the result of an presence detection performed by processor 102. The conditions used to generate the signal CTRL can vary depending on the design criteria of a particular implementation.

[0078] A multi-step approach can be implemented to activate and / or disable the capture device 104 based on the output of the motion sensor 164 and / or any other power consumption characteristics of the camera system 100, thereby reducing the power consumption of the camera system 100 and extending the lifespan of the battery 152. The motion sensor in sensor 164 may have low power consumption on the battery 152 (e.g., less than 10W). In the example, the motion sensor in sensor 164 may be configured to remain on (e.g., always active) unless disabled in response to feedback from the processor / SoC 102. Video analysis performed by the processor / SoC 102 may have relatively high power consumption on the battery 152 (e.g., greater than that of the motion sensor 164). In the example, the processor / SoC 102 may be in a low-power state (or powered down) until some motion is detected by the motion sensor in sensor 164.

[0079] Camera system 100 can be configured to operate using various power states. For example, in a power-down state (e.g., sleep state, low power state), the motion sensor in sensor 164 and processor / SoC 102 can be turned on, and other components of camera system 100 (e.g., image capture device 106, memory 150, communication module 154, etc.) can be turned off. In another example, camera system 100 can operate in an intermediate state. In the intermediate state, image capture device 106 can be turned on, and memory 150 and / or communication module 154 can be turned off. In yet another example, camera system 100 can operate in a power-on (or high-power) state. In the power-on state, sensor 164, processor / SoC 102, capture device 104, memory 150, and / or communication module 154 can be turned on. Camera system 100 can consume some power from battery 152 (e.g., a relatively small and / or minimal amount of power) in the power-down state. In the power-on state, camera system 100 can consume more power from battery 152. When the camera system 100 operates in each of the power states, the power state and / or the number of components turned on by the camera system 100 can vary according to the design criteria of a particular implementation.

[0080] In some embodiments, the camera system 100 may be implemented as a system-on-a-chip (SoC). For example, the camera system 100 may be implemented as a printed circuit board including one or more components. The camera system 100 may be configured to perform intelligent video analysis on video frames of a video. The camera system 100 may be configured to crop and / or enhance the video.

[0081] In some embodiments, the video frame may be a view (or a derivative of a view) captured by the capture device 104. Pixel data signals may be enhanced by the processor 102 (e.g., color conversion, noise filtering, automatic exposure, automatic white balance, automatic focus, etc.). In some embodiments, the video frame may provide a series of cropped and / or enhanced video frames that improve the view from the perspective of the camera system 100 (e.g., providing night vision, providing high dynamic range (HDR) imaging, providing more viewing area, highlighting detected objects, providing additional data (e.g., digital distance to the detected object), etc.) to enable the processor 102 to see the location better than a human can see with human vision.

[0082] Encoded video frames can be processed locally. In one example, the encoded video can be stored locally by memory 150, enabling processor 102 to facilitate computer vision analysis internally (e.g., without first uploading the video frames to a cloud service). Processor 102 can be configured to select video frames to be encapsulated into a video stream that can be transmitted over a network (e.g., a bandwidth-limited network).

[0083] In some embodiments, processor 102 may be configured to perform sensor fusion operations. The sensor fusion operations performed by processor 102 may be configured to analyze information from multiple sources (e.g., capture device 104, sensor 164, and HID 166). By analyzing various data from different sources, sensor fusion operations may be able to make inferences about the data that might not be possible from just one data source. For example, sensor fusion operations implemented by processor 102 may analyze video data (e.g., mouth movements) as well as speech patterns from directional audio. Different sources can be used to develop scene models to support decision-making. For example, processor 102 may be configured to compare the synchronization of detected speech patterns with mouth movements in video frames to determine which person is speaking in the video frame. Sensor fusion operations may also provide temporal correlation, spatial correlation, and / or reliability between the received data.

[0084] In some embodiments, processor 102 may implement convolutional neural network (CNN) capabilities. CNN capabilities can be implemented using deep learning techniques for computer vision. CNN capabilities can be configured to perform pattern and / or image recognition using a training process involving multi-layer feature detection. Computer vision and / or CNN capabilities can be executed locally by processor 102. In some embodiments, processor 102 may receive training data and / or feature set information from external sources. For example, external devices (e.g., cloud services) may access various data sources to provide training data that camera system 100 may not be able to obtain. However, computer vision operations performed using feature sets can be performed using the computational resources of processor 102 within camera system 100.

[0085] The video pipeline of processor 102 can be configured to perform local dewarping, cropping, enhancement, rolling shutter correction, stabilization, downsizing, encapsulation, compression, conversion, mixing, synchronization, and / or other video operations. The video pipeline of processor 102 can support multiple streams (e.g., generating multiple bitstreams in parallel, each with a different bitrate). In the example, the video pipeline of processor 102 can implement an image signal processor (ISP) with an input pixel rate of 320 Mbps. The architecture of the video pipeline of processor 102 enables real-time and / or near-real-time video operations on high-resolution video and / or high-bitrate video data. The video pipeline of processor 102 can perform computer vision processing, stereo vision processing, object detection, 3D noise reduction, fisheye lens correction (e.g., real-time 360-degree dewarping and lens distortion correction), oversampling, and / or high dynamic range processing on 4K resolution video data. In one example, the architecture of the video pipeline can achieve 4K ultra-high resolution with H.264 encoding at twice the real-time speed (e.g., 60fps) and 4K ultra-high resolution with H.265 / HEVC and / or 4K AVC encoding at 30fps (e.g., 4KP30 AVC and HEVC encoding with multi-stream support). The type of video operation and / or the type of video data operated on by the processor 102 can vary according to the design criteria of the specific implementation.

[0086] The camera sensor 180 can be a high-resolution sensor. Using the high-resolution sensor 180, the processor 102 can combine oversampling of the image sensor 180 with digital scaling within the cropped area. Oversampling and digital scaling can each be one of the video operations performed by the processor 102. Oversampling and digital scaling can be implemented to provide a higher resolution image within the total size limit of the cropped area.

[0087] In some embodiments, lens 160 may be a fisheye lens. One of the video operations implemented by processor 102 may be a de-distortion operation. Processor 102 may be configured to de-distort generated video frames. De-distortion may be configured to reduce and / or remove severe distortion caused by fisheye lens and / or other lens characteristics. For example, de-distortion may reduce and / or eliminate bulging effects to provide a linear image.

[0088] Processor 102 can be configured to crop (e.g., trim) a region of interest from a full video frame (e.g., generate a region of interest video frame). Processor 102 can generate video frames and select regions. In the example, cropping the region of interest can generate a second image. The cropped image (e.g., the region of interest video frame) can be smaller than the original video frame (e.g., the cropped image can be a portion of the captured video).

[0089] The region of interest (ROI) can be dynamically adjusted based on the location of the audio source. For example, the detected audio source may be moving, and its location may shift as video frames are captured. Processor 102 can update the coordinates of the selected ROI and dynamically update the cropped portion (e.g., a directional microphone implemented as one or more sensors in sensor 164 can dynamically update its location based on captured directional audio). The cropped portion may correspond to the selected ROI. As the ROI changes, the cropped portion may change. For example, the selected coordinates of the ROI may change frame by frame, and processor 102 may be configured to crop the selected region in each frame.

[0090] Processor 102 can be configured to oversample image sensor 180. Oversampling of image sensor 180 can produce a higher resolution image. Processor 102 can be configured to digitally upscale regions of video frames. For example, processor 102 can digitally upscale a cropped region of interest. For example, processor 102 can establish a region of interest based on directional audio, crop the region of interest, and then digitally upscale the cropped region of interest video frame.

[0091] The de-distortion operation performed by processor 102 can adjust the visual content of the video data. The adjustment performed by processor 102 can make the visual content appear natural (e.g., as if seen by a person viewing a position corresponding to the field of view of capture device 104). In the example, de-distortion can alter the video data to generate linear video frames (e.g., correcting artifacts caused by lens characteristics of lens 160). De-distortion operations can be implemented to correct distortions caused by lens 160. Adjusted visual content can be generated to achieve more accurate and / or reliable object detection.

[0092] Various features (e.g., de-distortion, digital scaling, cropping, etc.) can be implemented as hardware modules in processor 102. Implementing hardware modules can increase the video processing speed of processor 102 (e.g., faster than software implementations). Hardware implementations enable video processing while reducing latency. The hardware components used can vary depending on the design standards of the specific implementation.

[0093] The processor 102 shown includes multiple blocks (or circuits) 190a-190n. Blocks 190a-190n can implement various hardware modules implemented by the processor 102. Hardware modules 190a-190n can be configured to provide various hardware components to implement a video processing pipeline. Circuits 190a-190n can be configured to receive pixel data VIDEO, generate video frames based on the pixel data, perform various operations on the video frames (e.g., de-distortion, rolling shutter correction, cropping, magnification, image stabilization, 3D reconstruction, presence detection, automatic exposure, etc.), prepare video frames for communication with external hardware (e.g., encoding, encapsulation, color correction, etc.), parse feature sets, and implement various computer vision operations (e.g., object detection, segmentation, classification, etc.). Hardware modules 190a-190n can be configured to implement various security features (e.g., secure boot, I / O virtualization, etc.). Various implementations of the processor 102 may not necessarily utilize all the features of hardware modules 190a-190n. The features and / or functions of hardware modules 190a-190n may vary according to design standards for a particular implementation. Details of hardware modules 190a-190n may be described in connection with U.S. Patent Application No. 16 / 831,549, filed April 16, 2020; U.S. Patent Application No. 16 / 288,922, filed February 28, 2019; U.S. Patent Application No. 15 / 593,493 (now U.S. Patent No. 10,437,600), filed May 12, 2017; U.S. Patent Application No. 15 / 931,942, filed May 14, 2020; U.S. Patent Application No. 16 / 991,344, filed August 12, 2020; and U.S. Patent Application No. 17 / 479,034, filed September 20, 2021, the appropriate portions of which are incorporated herein by reference in their entirety.

[0094] Hardware modules 190a-190n can be implemented as dedicated hardware modules. Compared to software implementation, using dedicated hardware modules 190a-190n to implement various functions of processor 102 allows processor 102 to be highly optimized and / or customized to limit power consumption, reduce heat generation, and / or increase processing speed. Hardware modules 190a-190n can be customizable and / or programmable to implement multiple types of operations. Implementing dedicated hardware modules 190a-190n allows the hardware used to perform each type of computation to be optimized for speed and / or efficiency. For example, hardware modules 190a-190n can implement several relatively simple operations frequently used in computer vision operations, which together enable computer vision operations to be performed in real time. The video pipeline can be configured to identify objects. Objects can be identified by interpreting numerical and / or symbolic information to determine the specific type and / or characteristics of the object represented by the visual data. For example, the number of pixels and / or the color of pixels in video data can be used to identify portions of video data as objects. Hardware modules 190a-190n enable computationally intensive operations (e.g., computer vision operations, video encoding, video transcoding, 3D reconstruction, depth map generation, presence detection, etc.) to be performed locally by the camera system 100.

[0095] One of the hardware modules 190a-190n (e.g., 190a) can implement a scheduler circuit. Scheduler circuit 190a can be configured to store a directed acyclic graph (DAG). In the example, scheduler circuit 190a can be configured to generate and store a DAG in response to received (e.g., loaded) feature set information. The DAG can define video operations to be performed to extract data from video frames. For example, the DAG can define various mathematical weights (e.g., neural network weights and / or biases) to be applied when performing computer vision operations to classify various groups of pixels into specific objects.

[0096] Scheduler circuit 190a can be configured to parse acyclic graphs to generate various operators. Operators can be scheduled by scheduler circuit 190a in one or more of the other hardware modules 190a-190n. For example, one or more hardware modules 190a-190n can implement a hardware engine configured to perform a specific task (e.g., a hardware engine designed to perform repetitive specific mathematical operations used for performing computer vision operations). Scheduler circuit 190a can schedule operators based on when they are ready to be processed by hardware engines 190a-190n.

[0097] Scheduler circuit 190a can time-multiplex tasks to hardware modules 190a-190n for execution based on the availability of hardware modules 190a-190n. Scheduler circuit 190a can parse a directed acyclic graph into one or more data streams. Each data stream can include one or more operators. Once the directed acyclic graph is parsed, scheduler circuit 190a can allocate data streams / operators to hardware engines 190a-190n and send relevant operator configuration information to initiate the operators.

[0098] The binary representation of each directed acyclic graph can be an ordered traversal of the directed acyclic graph, where descriptors and operators are interleaved based on data dependencies. Descriptors typically provide registers that link data buffers to specific operands in a dependent operator. In various embodiments, operators may not appear in the directed acyclic graph representation until all dependent descriptors have been declared for operands.

[0099] One of the hardware modules 190a-190n (e.g., 190b) can implement an Artificial Neural Network (ANN) module. The ANN module can be implemented as a fully connected neural network or a Convolutional Neural Network (CNN). In this example, the fully connected network is "structure-agnostic" because no special assumptions need to be made about the input. A fully connected neural network consists of a series of fully connected layers that connect each neuron in one layer to each neuron in another layer. In a fully connected layer, for n inputs and m outputs, there are n*m ​​weights. There are also bias values ​​for each output node, resulting in a total of (n+1)*m parameters. In a trained neural network, (n+1)*m parameters have been determined during the training process. A trained neural network typically includes an architectural specification and a set of parameters (weights and biases) determined during training. In another example, a CNN architecture may explicitly assume that the input is an image, enabling the encoding of specific attributes into the model architecture. A CNN architecture may include a sequence of layers, where each layer transforms one activation into another through a differentiable function.

[0100] In the example shown, the artificial neural network 190b can implement a convolutional neural network (CNN) module. The CNN module 190b can be configured to perform computer vision operations on video frames. The CNN module 190b can be configured to achieve object recognition through multi-layer feature detection. The CNN module 190b can be configured to compute descriptors based on the performed feature detections. The descriptors enable the processor 102 to determine the probability that a pixel in a video frame corresponds to a specific object (e.g., a specific manufacture / model / year of a vehicle, identifying a person as a specific individual, detecting the type of animal, detecting facial features, etc.).

[0101] CNN module 190b can be configured to implement convolutional neural network capabilities. CNN module 190b can be configured to use deep learning techniques to implement computer vision. CNN module 190b can be configured to use a training process through multi-layer feature detection to implement pattern and / or image recognition. CNN module 190b can be configured to perform inference against machine learning models.

[0102] The CNN module 190b can be configured to perform feature extraction and / or matching solely in hardware. Feature points typically represent regions of interest (e.g., corners, edges, etc.) in a video frame. By temporarily tracking feature points, estimates of the self-motion of the capture platform or motion models of objects observed in a scene can be generated. To track feature points, a matching operation is typically incorporated into the CNN module 190b by hardware to find the most probable correspondence between feature points in a reference video frame and a target video frame. During the matching of reference and target feature points, each feature point can be represented by a descriptor (e.g., image patch, SIFT, BRIEF, ORB, FREAK, etc.). Implementing the CNN module 190b using dedicated hardware circuitry allows for real-time computation of descriptor matching distances.

[0103] The CNN module 190b can be configured to perform face detection, face recognition, and / or presence determination. For example, face detection, face recognition, and / or presence determination can be performed based on a trained neural network implemented by the CNN module 190b. In some embodiments, the CNN module 190b can be configured to generate a depth image based on a structured light pattern. The CNN module 190b can be configured to perform various detection and / or recognition operations and / or perform 3D recognition operations.

[0104] CNN module 190b can be a dedicated hardware module configured to perform feature detection on video frames. Features detected by CNN module 190b can be used to compute descriptors. CNN module 190b can determine, in response to the descriptors, the probability that a pixel in a video frame belongs to a specific object and / or multiple objects. For example, using the descriptors, CNN module 190b can determine the probability that a pixel corresponds to a specific object (e.g., a person, furniture item, pet, vehicle, etc.) and / or characteristics of the object (e.g., the shape of the eyes, the distance between facial features, the hood of a vehicle, body parts, a vehicle's license plate, a person's face, clothing worn by a person, etc.). Implementing CNN module 190b as a dedicated hardware module of processor 102 allows device 100 to perform computer vision operations locally (e.g., on-chip) without relying on the processing power of remote devices (e.g., communicating data to a cloud computing service).

[0105] The computer vision operations performed by the CNN module 190b can be configured to perform feature detection on video frames to generate descriptors. The CNN module 190b can perform object detection to determine regions in the video frames with a high probability of matching a specific object. In one example, an open operand stack can be used to customize the type of the objects(s) to be matched (e.g., reference objects) (thus enabling the programmability of processor 102 to implement various artificial neural networks defined by directed acyclic graphs, each providing instructions for performing various types of object detection). The CNN module 190b can be configured to perform local masking on regions with a high probability of matching a specific object(s) to detect the object.

[0106] In some embodiments, the CNN module 190b can determine the location (e.g., 3D coordinates and / or positional coordinates) of various features (e.g., characteristics) of a detected object. In one example, 3D coordinates can be used to determine the position of a person's arms, legs, chest, and / or eyes. A positional coordinate for the vertical positioning of a body part in 3D space on a first axis and another coordinate for the horizontal positioning of a body part in 3D space on a second axis can be stored. In some embodiments, the distance from the lens 160 can represent a coordinate for the depth positioning of a body part in 3D space (e.g., a positional coordinate on a third axis). Using the positioning of the various body parts in 3D space, the processor 102 can determine the body position and / or the detected human body characteristics.

[0107] The CNN module 190b can be pre-trained (e.g., configured to perform computer vision to detect objects based on received training data). For example, the results of the training data (e.g., a machine learning model) can be pre-programmed and / or loaded into processor 102. The CNN module 190b can perform inference against the machine learning model (e.g., to perform object detection). Training can include determining weight values ​​for each layer of the neural network model. For example, weight values ​​can be determined for each of the layers used for feature extraction (e.g., convolutional layers) and / or for classification (e.g., fully connected layers). The weight values ​​learned by the CNN module 190b can vary depending on the design criteria of a particular implementation.

[0108] The CNN module 190b can perform feature extraction and / or object detection by performing convolution operations. These convolution operations can be hardware-accelerated for fast (e.g., real-time) computation that can be performed with low power consumption. In some embodiments, the convolution operations performed by the CNN module 190b can be used to perform computer vision operations. In some embodiments, the convolution operations performed by the CNN module 190b can be used for any function (e.g., 3D reconstruction) that may involve computing convolution operations and is performed by the processor 102.

[0109] Convolution operations can include sliding a feature detection window along a layer while performing computations (e.g., matrix operations). The feature detection window can apply filters to pixels and / or extract features associated with each layer. The feature detection window can be applied to a pixel and multiple surrounding pixels. In the example, a layer can be represented as a matrix representing the values ​​of features from one of the pixels and / or layers, and the filters applied by the feature detection window can be represented as matrices. Convolution operations can apply matrix multiplication between regions of the current layer covered by the feature detection window. Convolution operations can cause the feature detection window to slide along a region of the layer to generate a result representing each region. The size of the regions, the type of operation applied by the filters, and / or the number of layers can vary depending on the design criteria of a particular implementation.

[0110] Using convolutional operations, the CNN module 190b can compute multiple features for pixels of the input image at each extraction step. For example, each layer in a layer can receive input from a set of features located in a small neighborhood (e.g., a region) of a previous layer (e.g., a local receptive field). Convolutional operations can extract basic visual features (e.g., oriented edges, endpoints, corners, etc.), which are then combined by higher layers. Since the feature extraction window operates on pixels and nearby pixels (or subpixels), the result of the operation can have location invariance. Layers can include convolutional layers, pooling layers, non-linear layers, and / or fully connected layers. In the example, convolutional operations can learn to detect edges from raw pixels (e.g., the first layer), then use features from previous layers (e.g., detected edges) to detect shapes in the next layer, and then use the shapes to detect higher-level features in higher layers (e.g., facial features, pets, vehicles, vehicle components, furniture, etc.), and the last layer can be a classifier using the higher-level features.

[0111] The CNN module 190b can perform data streams directed to feature extraction and matching, including two-stage detection, warping operators, component operators manipulating component lists (e.g., components can be regions of vectors sharing common attributes and can be combined with bounding boxes), matrix inversion operators, dot product operators, convolution operators, conditional operators (e.g., multiplexing and demultiplexing), remapping operators, min-max reduction operators, pooling operators, non-minimum non-maximum suppression operators, scan-window-based non-maximum suppression operators, aggregation operators, scattering operators, statistical operators, classifier operators, integral image operators, comparison operators, indexing operators, pattern matching operators, feature extraction operators, feature detection operators, two-stage object detection operators, score generation operators, block reduction operators, and upsampling operators. The types of operations performed by the CNN module 190b to extract features from training data can vary depending on the design criteria of a particular implementation.

[0112] Each of the hardware modules 190a-190n can implement a processing resource (or hardware resource or hardware engine). Hardware engines 190a-190n can be operable to perform specific processing tasks. In some configurations, hardware engines 190a-190n can operate in parallel and independently of each other. In other configurations, hardware engines 190a-190n can operate collaboratively to perform assigned tasks. One or more of the hardware engines 190a-190n can be homogeneous processing resources (all circuits 190a-190n can have the same capabilities) or heterogeneous processing resources (two or more circuits 190a-190n can have different capabilities).

[0113] refer to Figure 4 The diagram illustrates the processing circuitry of a camera system 100 configured to perform object tracking and sorting using a convolutional neural network. In this example, the processing circuitry of the camera system 100 can be configured for a variety of applications, including but not limited to: autonomous and semi-autonomous vehicles (e.g., cars, trucks, motorcycles, agricultural machinery, drones, aircraft, etc.), manufacturing, and / or security and surveillance systems. Compared to a general-purpose computer, the processing circuitry of the camera system 100 typically comprises hardware circuitry optimized to provide high-performance image processing and computer vision pipelines in a minimal area and with minimal power consumption. In this example, various operations for performing image processing, feature detection / extraction, 3D reconstruction, presence detection, depth map generation, and / or object detection / classification for computer (or machine) vision can be implemented using hardware modules designed to reduce computational complexity and utilize resources efficiently.

[0114] In an example embodiment, processing circuitry 100 may include processor 102, memory 150, general-purpose processor 158, and / or memory bus 200. General-purpose processor 158 may implement a first processor. Processor 102 may implement a second processor. In this example, circuitry 102 may implement a computer vision processor. In this example, processor 102 may be an intelligent vision processor. Memory 150 may implement external memory (e.g., memory outside of circuitry 158 and 102). In this example, circuitry 150 may be implemented as dynamic random access memory (DRAM) circuitry. The processing circuitry of camera system 100 may include other components (not shown). The number, type, and / or arrangement of components in the processing circuitry of camera system 100 may vary according to design criteria for a particular implementation.

[0115] The general-purpose processor 158 may be operable to interact with circuits 102 and 150 to perform various processing tasks. In an example, processor 158 may be configured as a controller for circuit 102. Processor 158 may be configured to execute computer-readable instructions. In one example, the computer-readable instructions may be stored by circuit 150. In some embodiments, the computer-readable instructions may include controller operations. Processor 158 may be configured to communicate with circuit 102 and / or access results generated by components of circuit 102. In an example, processor 158 may be configured to utilize circuit 102 to perform operations associated with one or more neural network models.

[0116] In the example, processor 102 typically includes scheduler circuitry 190a, block (or circuitry) 202, one or more blocks (or circuitries) 204a-204n, block (or circuitry) 206, and path 208. Block 202 may implement a directed acyclic graph (DAG) memory. DAG memory 202 may include a CNN module 190b and / or weighted / bias values ​​210. Blocks 204a-204n may implement hardware resources (or engines). Block 206 may implement shared memory circuitry. In the example embodiment, one or more circuits in circuits 204a-204n may include blocks (or circuitries) 212a-212n. In the illustrated example, circuits 212a and 212b are implemented as representative examples of the corresponding hardware engines 204a-204b. One or more circuits in circuits 202, 204a-204n, and / or circuit 206 may be related to... Figure 3 Example implementations of hardware modules 190a-190n are shown in association.

[0117] In the example, processor 158 may be configured to program circuit 102 using one or more pre-trained artificial neural network models (ANNs), including a convolutional neural network (CNN) 190b having multiple output frames according to embodiments of the invention and a weighted / kernel (WGTS) 210 utilized by the CNN module 190b. In various embodiments, the CNN module 190b may be configured (trained) for operation in an edge device. In the example, the processing circuitry of camera system 100 may be coupled to a sensor (e.g., a video camera, etc.) configured to generate data input. The processing circuitry of camera system 100 may be configured to generate one or more outputs in response to data input from the sensor based on one or more inferences made by executing the pre-trained CNN module 190b and the weighted / kernel (WGTS) 210. The operations performed by processor 158 may vary depending on the design criteria of a particular implementation.

[0118] In various embodiments, circuit 150 may implement dynamic random access memory (DRAM) circuitry. Circuit 150 is typically an operable multidimensional array for storing input data elements and various forms of output data elements. Circuit 150 may exchange input and output data elements with processor 158 and processor 102.

[0119] Processor 102 may implement computer vision processor circuitry. In examples, processor 102 may be configured to implement various functions for computer vision. Processor 102 is typically operable to perform specific processing tasks arranged by processor 158. In various embodiments, all or part of processor 102 may be implemented individually in hardware. Processor 102 may directly execute data streams involving the execution of CNN module 190b and generated by software (e.g., directed acyclic graphs, etc.) for specified processing tasks (e.g., computer vision, 3D reconstruction, presence detection, etc.). In some embodiments, processor 102 may be a representative example of multiple computer vision processors implemented by the processing circuitry of camera system 100 and configured to operate together.

[0120] In the example, circuit 212a can implement a convolution operation. In another example, circuit 212b can be configured to provide a dot product operation. Convolution and dot product operations can be used to perform computer (or machine) vision tasks (e.g., as part of an object detection process). In yet another example, one or more of the circuits 204c-204n may include blocks (or circuits) 212c-212n (not shown) for providing multidimensional convolution computation. In yet another example, one or more of the circuits 204a-204n can be configured to perform a 3D reconstruction task.

[0121] In the example, circuit 102 can be configured to receive a directed acyclic graph (DAG) from processor 158. The DAG received from processor 158 can be stored in DAG memory 202. Circuit 102 can be configured to perform the DAG for CNN module 190b using circuits 190a, 204a-204n, and 206.

[0122] Multiple signals (e.g., OP_A-OP_N) can be exchanged between circuit 190a and corresponding circuits 204a-204n. Each signal in OP_A-OP_N can convey operation execution information and / or operation generation information. Multiple signals (e.g., MEM_A to MEM_N) can be exchanged between corresponding circuits 204a-204n and circuit 206. Signals MEM_A to MEM_N can carry data. Signals (e.g., DRAM) can be exchanged between circuit 150 and circuit 206. Signal DRAM can transmit data between circuits 150 and 190a (e.g., on transmission path 208).

[0123] Scheduler circuit 190a is typically operable to schedule tasks between circuits 204a-204n to perform various computer vision-related tasks defined by processor 158. Individual tasks can be assigned by scheduler circuit 190a to circuits 204a-204n. Scheduler circuit 190a can assign individual tasks in response to parsing a directed acyclic graph (DAG) provided by processor 158. Scheduler circuit 190a can time-multiplex tasks between circuits 204a-204n based on their availability for performing work.

[0124] Each circuit 204a-204n can implement a processing resource (or hardware engine). Hardware engines 204a-204n are typically operable to perform a specific processing task. Hardware engines 204a-204n can be implemented as including dedicated hardware circuitry optimized for high performance and low power consumption while performing the specific processing task. In some configurations, hardware engines 204a-204n can operate in parallel and independently of each other. In other configurations, hardware engines 204a-204n can operate collaboratively with each other to perform assigned tasks.

[0125] Hardware engines 204a-204n can be homogeneous processing resources (e.g., all circuits 204a-204n can have the same capabilities) or heterogeneous processing resources (e.g., two or more circuits 204a-204n can have different capabilities). Hardware engines 204a-204n are typically configured to execute operators, which can include, but are not limited to, resampling operators, warping operators, component operators that manipulate a list of components (e.g., components can be regions of vectors sharing common attributes and can be combined with bounding boxes), matrix inversion operators, dot product operators, convolution operators, conditional operators (e.g., multiplexing and demultiplexing), remapping operators, min-max reduction operators, pooling operators, non-minimum non-maximum suppression operators, aggregation operators, scattering operators, statistical operators, classifier operators, integral image operators, upsampling operators, and power-of-two downsampling operators, etc.

[0126] In the example, hardware engines 204a-204n may include matrices stored in various memory buffers. The matrices stored in the memory buffers can be used to initialize convolution operators. The convolution operators can be configured to efficiently perform computations that are repeatedly performed on a convolution function. In the example, hardware engines 204a-204n implementing the convolution operators may include multiple mathematical circuits configured to process multi-bit input values ​​and operate in parallel. Convolution operators can provide an efficient and general solution for computer vision and / or 3D reconstruction by using one-dimensional or higher-dimensional kernels to compute convolutions (also known as cross-correlation). Convolutions can be useful in computer vision operations such as object detection, object recognition, edge enhancement, image smoothing, etc. The techniques and / or architectures implemented by this invention can be operable to compute convolutions of input arrays with kernels. Details of the convolution operators can be described in connection with U.S. Patent No. 10,310,768, filed January 11, 2017, the appropriate portions of which are incorporated herein by reference.

[0127] In various embodiments, hardware engines 204a-204n may be implemented as hardware circuits only. In some embodiments, hardware engines 204a-204n may be implemented as general-purpose engines, which may be configured to operate as dedicated machines (or engines) through circuit customization and / or software / firmware. In some embodiments, hardware engines 204a-204n may alternatively be implemented as one or more instances or threads of program code executing on processor 158 and / or one or more processors 102 (including, but not limited to, vector processors, central processing units (CPUs), digital signal processors (DSPs), or graphics processing units (GPUs)). In some embodiments, scheduler 190a may select one or more of hardware engines 204a-204n for a specific process and / or thread. Scheduler 190a may be configured to assign hardware engines 204a-204n to specific tasks in response to resolving a directed acyclic graph stored in DAG memory 202.

[0128] Circuit 206 can implement shared memory circuitry. Shared memory 206 can be configured to store data in response to input requests and / or present data in response to output requests (e.g., requests from processor 158, DRAM 150, scheduler circuitry 190a, and / or hardware engines 204a-204n). In this example, shared memory circuitry 206 can be implemented as on-chip memory for computer vision processor 102. Shared memory 206 is typically operable to store all or part of a multidimensional array (or vector) of input and output data elements generated and / or utilized by hardware engines 204a-204n. Input data elements can be transferred from DRAM circuitry 150 to shared memory 206 via memory bus 200. Output data elements can be sent from shared memory 206 to DRAM circuitry 150 via memory bus 200.

[0129] Path 208 can implement a transfer path within processor 102. Transfer path 208 is typically operable to move data from scheduler circuitry 190a to shared memory 206. Transfer path 208 can also be operable to move data from shared memory 206 to scheduler circuitry 190a.

[0130] Processor 158 is shown communicating with computer vision processor 102. Processor 158 can be configured as a controller for computer vision processor 102. In some embodiments, processor 158 can be configured to transmit instructions to scheduler 190a. For example, processor 158 can provide one or more directed acyclic graphs to scheduler 190a via DAG memory 202. Scheduler 190a can initialize and / or configure hardware engines 204a-204n in response to parsing the directed acyclic graphs. In some embodiments, processor 158 can receive status information from scheduler 190a. For example, scheduler 190a can provide status information and / or output readiness from hardware engines 204a-204n to processor 158 so that processor 158 can determine one or more next instructions to execute and / or decisions to make. In some embodiments, processor 158 may be configured to communicate with shared memory 206 (e.g., directly or via scheduler 190a, which receives data from shared memory 206 via path 208). Processor 158 may be configured to retrieve information from shared memory 206 to make decisions. The instructions executed by processor 158 in response to information from computer vision processor 102 may vary depending on the design criteria of a particular implementation.

[0131] refer to Figure 5 A block diagram illustrating an intelligent automatic exposure metering system is shown. A dynamic AI metering component 300 is shown. The dynamic AI metering component 300 may include various hardware, concept blocks, inputs, and / or outputs that can be used by the device 100 to perform AI metering techniques for automatic exposure. The dynamic AI metering component 300 can be represented as a block diagram illustrating operations performed by the device 100 to determine regions of interest (ROIs) and / or generate AE parameters for various subjects.

[0132] The dynamic AI measurement component 300 may include a processor 102, video frames 302a-302n, block (or circuit) 304, block (or circuit) 306, and / or block (or circuit) 308. Circuit 304 may implement an image digital signal processor (IDSP). Circuit 306 may implement a region of interest (ROI) detection module. Circuit 308 may implement a 3A module. For illustrative purposes, the processor 102, IDSP 304, ROI detection module 306, and 3A module 308 are shown as separate components. However, the IDSP 304, ROI detection module 306, and 3A module 308 may each be a component implemented by the processor 102. The dynamic AI measurement component 300 may include other components (not shown). The number, type, and / or arrangement of the components of the dynamic AI measurement component 300 may vary according to the design criteria of a particular implementation.

[0133] Processor 102 can be configured to receive a signal VIDEO. The signal VIDEO may include RGB pixel data generated by image sensor 180. The pixel data may include captured information about the environment and / or objects near capture device 104. In some embodiments, the pixel data may include RGB data and a structured light pattern projected onto the environment and / or objects. Processor 102 can generate signals (e.g., FRAMES). The signal FRAMES may include video frames 302a-302n. Processor 102 can be configured to process the pixel data arranged as video frames 302a-302n. Video frames 302a-302n may be presented to CNN module 190b (e.g., processed internally by processor 102 using CNN module 190b). Processor 102 is capable of performing other operations using video frames 302a-302n (e.g., generating encoded video frames for display, encapsulating video frames 302a-302n for communication using communication module 154, etc.).

[0134] IDSP 304 can be configured to receive video frames 302a-302n. IDSP 304 can be configured to process the raw video frames 302a-302n one frame at a time. IDSP 304 can be configured to generate signals (e.g., TS), signals (e.g., 3A_STAT), and / or signals (e.g., YUV). Signal TS may include a timestamp of one of the video frames 302a-302n processed by IDSP 304. Signal 3A_STAT may include 3A statistics generated by IDSP 304 in response to one of the video frames 302a-302n. Signal YUV may include a YUV image extracted by IDSP 304 from one of the video frames 302a-302n. IDSP 304 can be configured to generate and / or receive other data (not shown). The amount and / or type of data inputs received by IDSP 304 and / or data outputs generated by IDSP 304 may vary depending on the design criteria of the specific implementation.

[0135] In response to analyzing video frames 302a-302n one at a time, a timestamp signal TS and a statistical data signal 3A_STAT can be extracted and / or generated. The 3A statistical data can be output data from the IDSP 304, which may include multiple data entries, such as luminance and / or histograms. For example, a table of tile luminance values ​​may be one data entry from the 3A statistical data. In another example, the 3A statistical data may include information about automatic white balance, automatic focus, and automatic exposure. The 3A statistical data may include a luminance value for each tile, having a value range of [0, 4095], including extreme values. The timestamp signal TS and the statistical data signal 3A_STAT can be presented to the 3A module 308. The IDSP 304 can be configured to forward the YUV image along with the timestamp signal TS to the ROI detection module 306.

[0136] The ROI detection module 306 may be a component of the CNN module 190b. The ROI detection module 306 may be configured to perform various computer vision operations on YUV images generated from video frames 302a-302n. The ROI detection module 306 may receive a timestamp signal TS and a signal YUV. The ROI detection module 306 may be configured to generate a signal (e.g., TS'), a signal (e.g., FN), and a signal (e.g., LROI). The signal TS' may include an updated timestamp. The signal FN may include the frame number of one of the video frames 302a-302n analyzed by the ROI detection module 306. In some embodiments, the YUV image may communicate with the signal FN. The signal LROI may include a list of entries for the region of interest and / or information about the region of interest. The updated timestamp signal TS', the frame number signal FN, and / or the signal LROI may be generated by the ROI detection module 306 in response to the timestamp signal TS and / or the signal YUV. The updated timestamp signal TS', frame number signal FN, and / or signal LROI can be presented to the 3A module 308. The quantity and / or type of data input received by the ROI detection module 306 and / or data output generated by the ROI detection module 306 can vary according to the design criteria of a specific implementation.

[0137] The ROI detection module 306 can be configured to implement one or more computer vision AI models. These models can be configured to detect various objects, subjects, and / or events of interest in the YUV images generated from video frames 302a-302n. The computer vision AI models(s) implemented by the ROI detection module 306 can be configured to detect one or more types of objects and / or classify them. In one example, the computer vision AI model implemented by the ROI detection module 306 can be configured to perform person detection. In another example, the model can be configured to perform vehicle detection. In yet another example, the model can be configured to perform motion detection. In still another example, the model can be configured to perform face detection. In some embodiments, the model can be configured to perform a combination of various subject detections (e.g., detecting pedestrians and vehicles). The number of computer vision AI models and / or the types of objects detected as regions of interest in the YUV images of video frames 302a-302n can vary depending on the design criteria of a particular implementation.

[0138] The ROI detection module 306 can be configured to calculate each ROI within the YUV image of one video frame (302a-302n) at a time. The ROI detection module 306 can be configured to generate a timestamp (e.g., a modified timestamp) for the analyzed YUV image of one video frame (302a-302n). The ROI detection module 306 can also be configured to generate a frame number for the analyzed YUV image of one video frame (302a-302n). The calculated ROI information may include a list of regions of interest (e.g., a list of each calculated ROI corresponding to each object / subject). The list of ROIs may include multiple entries and / or elements (e.g., subject), a single element / entry, or even zero elements (e.g., no object of interest detected). The ROI information for each calculated ROI in the ROI list may include the ROI location (e.g., x, y coordinates in the YUV image), the ROI ID, and / or the ROI weighting. For example, the signal LROI can include a list of detected ROIs, which includes the ROI location, ROI ID, and ROI weight for each ROI.

[0139] A Region of Interest (ROI) location defines the start and end points of a region of interest within a YUV image. For example, an ROI location can include the x and y coordinates of a bounding box defining a detected subject. The size of the ROI location can depend on the size of the detected object relative to the size of the YUV image.

[0140] The ROI ID may include a unique number. A unique ROI ID can be assigned to each ROI to uniquely identify the detected object. The ROI ID can be used to enable the processor 102 to track the same object across one or more video frames in video frames 302a-302n. Each detected unique object can share the same ROI ID across video frames 302a-302n, and different objects can each have a different ROI ID number assigned by the ROI detection module 306.

[0141] ROI weighting can include floating-point numbers. In the example, ROI weighting can include values ​​between 0.0 and 1.0, including the endpoints. ROI weighting can be applied to each ROI region. ROI weighting can specify the priority of each ROI. Priority can be used to determine which ROI is likely to be preferred in terms of optimal exposure. For example, the ROI with the highest ROI weighting can receive the highest priority in terms of exposure, while the ROI with the lowest ROI weighting can receive the lowest priority. The sum of the ROI weights of all ROIs cannot exceed 1.0. The amount of ROI weighting applied to each detected object / subject can be determined by the implemented computer vision AI model(s). In some embodiments, the implemented computer vision AI model can detect people, faces, and vehicles, and the computer vision AI model can be configured to provide the highest ROI weighting to faces, then the ROI weighting to people, and then the ROI weighting to vehicles (e.g., to ensure that faces are identifiable for facial recognition while still providing exposure for other objects of interest (e.g., vehicles)). The amount of ROI weighting applied to each ROI can vary depending on the design criteria of a particular implementation.

[0142] A modified (or updated) timestamp (e.g., modified timestamp signal TS) generated by the ROI detection module 306 can be generated based on the timestamp (e.g., timestamp signal TS) of the YUV image from one of the video frames 302a-302n. For example, a timestamp can be assigned to a YUV image based on the timestamp signal TS received from the IDSP module 304. Typically, the modified timestamp signal TS' generated by the ROI detection module 306 may differ from the latest timestamp in the timestamp signal TS. For example, the computer vision analysis performed by the ROI detection module 306 may take a short time to complete. The ROI detection module 306 can perform analysis on a YUV image from video frames 302a-302n (e.g., video frame 302a), while the IDSP 304 provides a new timestamp from video frames 302a-302n (e.g., video frame 302b) for the next YUV image. If the computer vision operation performed by the ROI detection module 306 can be performed before the next video frame is received, the signal TS' may include a timestamp equal to the value of the input timestamp TS. The difference between the value in the signal TS and the value in the modified output signal TS' may depend on the computation speed of the computer vision operation performed by the ROI detection module 306.

[0143] The 3A module 308 can be configured to receive a timestamp signal TS, a statistical data signal 3A_STAT, a modified timestamp signal TS', a frame number signal FN, a signal LROI, a signal (e.g., KNEE), a signal (e.g., TLN), a signal (e.g., MWT), and / or a signal (e.g., TLUMA). The 3A module 308 can be configured to generate a parameter signal AE_PARAM. The signal AE_PARAM can be presented to the capture device 104. The quantity and / or type of data inputs received by the 3A module 308 and / or data outputs generated by the 3A module 308 can vary according to the design criteria of a particular implementation.

[0144] Module 308 may include block (or circuit) 320, block (or circuit) 322, and / or block (or circuit) 324. Circuit 320 may implement the AI ​​metering control module. Circuit 322 may implement the automatic exposure (AE) control module. Circuit 324 may implement the tracking list queue. Module 308 may include other components (not shown). The number, type, and / or arrangement of the components of module 308 may vary according to the design criteria of a particular implementation.

[0145] AI metering control module 320 can be configured to receive statistical data signal 3A_STAT and timestamp signal TS generated by IDSP 304, as well as modified timestamp signal TS', frame number signal FN, and signal LROI from ROI detection module 306. AI metering control module 320 can be configured to receive signal KNEE, signal TLN, and / or signal MWT. Signal KNEE, signal TLN, and / or signal MWT can be user-defined settings. In one example, one or more of signal KNEE, signal TLN, and / or signal MWT can be provided by signal USER from HID 166. In another example, signal KNEE, signal TLN, and / or signal MWT can be stored in memory 150. AI metering control module 320 can be configured to communicate signals (e.g., HTS) and signals (e.g., HLROI) with trace list queue 324. AI metering control module 320 can be configured to generate signals (e.g., CLUMA). Signal CLUMA can be presented to AE control module 322.

[0146] Signal TLN may include a tracking loop number. Signal KNEE may include a weighted value that balances exposure between bright and dark areas. Signal MWT may include a metering weighting table. The metering weighting table may define weighted values ​​for each tile (e.g., sub-region) of video frames 302a-302n. Signal TLN, signal KNEE, and / or signal MWT may include initial settings for the AI ​​metering control module 320.

[0147] The AI ​​metering control module 320 can be configured to evaluate the current luminance value of one of the video frames 302a-302n in response to received input. For example, the current luminance value can be calculated in response to the timestamp of one of the video frames 302a-302n, the updated timestamp TS' of one of the video frames 302a-302n, 3A statistics (e.g., tile luminance values), and a list of regions of interest. Furthermore, the AI ​​metering control module 320 can also evaluate the current luminance value based on exposure weighting, tracking loop number, and / or a metering weighting table. The signal CLUMA may include the current luminance value calculated by the AI ​​metering control module 320.

[0148] The AI ​​metering control module 320 can be configured to determine the operating mode for performing AI metering in response to signals HTS and / or HLROI. The HLROI signal may include historical region of interest list information (e.g., previous history of the region of interest) generated by the ROI detection module 306 (e.g., derived from data provided in the HLROI signal). The HTS signal may include historical timestamps for entries in the historical list of ROI information.

[0149] The AE control module 322 can be configured to receive signals CLUMA and TLUMA. The TLUMA signal may include a target luminance value. The AE control module 322 can be configured to generate a signal AE_PARAM. In response to the CLUMA signal and / or the TLUMA signal, the AE_PARAM signal can be presented to the capture device 104.

[0150] The AE control module 322 can be configured to generate AE parameters for the image sensor 180. The AE control module 322 can be configured to compare the current luminance value from the signal CLUMA with the target luminance value from the signal TLUMA. In response to the comparison between the signal TLUMA and the signal CLUMA, the AE control module 322 can calculate AE parameters (e.g., shutter speed, automatic gain control (AGC), aperture, etc.). For example, the shutter speed, AGC, and / or DC aperture value can be adjusted to achieve the target luminance. The target luminance value in the signal TLUMA can be a user-configurable value. In one example, the signal TLUMA can be provided by the signal USER from HID 166. In another example, the signal TLUMA can be provided by the memory 150.

[0151] The 3A module 308 can be configured to perform 3A techniques. The 3A techniques performed by the AI ​​metering control module 320 and AE control module 322 of the 3A module 308 can be performed independently of computer vision operations performed by the CNN module 190b (e.g., a computer vision AI model implemented by the ROI detection module 306). For example, the 3A techniques performed by the 3A module 308 can be performed in parallel with the computer vision operations performed by the ROI detection module 306. In one example, the ROI detection module can generate ROI information by analyzing the YUV images of video frames 302a-302n, while the AI ​​metering control module 320 analyzes the tile brightness values ​​from the 3A statistics of video frames 302a-302n. Communication between the ROI detection module 306 and the 3A module 308 can be achieved through inter-process communication (IPC) techniques. For example, IPC techniques may include message queues, sockets, shared memory, etc. The IDSP 304 and the 3A module 308 can communicate via shared memory. In the example, the shared memory used for the IDSP 304 and 3A module 308 can be the shared memory 206 of the processor 102. Data can be synchronized and retrieved via interrupts.

[0152] The tracking list queue 324 can be configured to store historical data about computer vision results generated by the ROI detection module 306. The tracking list queue 324 can be configured to communicate with the AI ​​measurement control module 320 the historical region of interest list signal HLROI and the historical timestamp signal HTS. For example, the data communicated in the historical region of interest list signal HLROI and the historical timestamp signal HTS may include historical computer vision results. The tracking list queue 324 can be configured as a memory queue (e.g., the tracking list queue 324 may include logic units for tracking ROIs).

[0153] The ROI detection module 306 can provide computer vision results (e.g., a region of interest list signal LROI and an updated timestamp signal TS') to the AI ​​measurement control module 320. The AI ​​measurement control module 320 can push new computer vision results to the tracking list queue 324 as historical computer vision results. For example, the AI ​​measurement control module 320 can generate a historical region of interest list signal HLROI in response to the region of interest list signal LROI, and generate a historical timestamp signal HTS in response to the updated timestamp signal TS'. When new data (e.g., in the historical ROI list signal HLROI and the updated timestamp signal HTS) is pushed from the AI ​​measurement control module 320 to the tracking list queue 324, the oldest data stored in the tracking list queue 324 can be pushed out. The tracking list queue 324 can have a predefined capacity of N recent computer vision results. In the example, if the tracking list queue has a capacity of three, the tracking list queue 324 can cache the last three sets of historical data (e.g., historical data A can be stored first, then historical data B, then historical data C, and when a new computer vision result (historical data D) is received, historical data A can be flushed and historical data D can be stored). The AI ​​econometrics control module 320 can use the historical data stored in the tracking list queue 324 for prediction (e.g., for cubic and / or quadratic fitting techniques) and / or for selecting which AI econometrics technique to use.

[0154] The AI ​​measurement technology executed by the AI ​​measurement control module 320 can be implemented in one of three different ways based on computer vision results selected from a list of regions of interest in the signal's LROI. For example, different implementation branches can be executed for AI measurement depending on whether there is no current computer vision result without history, no current computer vision result with history, and / or the current computer vision result.

[0155] If the signal LROI indicates that no computer vision result was received in the latest tracking loop number of video frames 302a-302n (e.g., based on the tracking loop number signal TLN), the AI ​​metrology control module 320 can operate in an operating mode that provides a metrology strategy for the entire field of view (FOV) of the current video frame. All tiles (e.g., sub-regions) can be considered in the current brightness calculation. The current brightness value CLUMA can be the weighted average brightness of the entire image. The calculation of the weighted average brightness for the entire input can be determined based on the signal KNEE (e.g., a weight that prioritizes exposure for dark or bright areas), the signal MWT (e.g., a metrology weighting table providing weights for each tile), and the signal 3A_STAT (e.g., the brightness value for each tile).

[0156] In the operating mode that implements the metering strategy for the entire FOV, the current brightness calculation may include two steps. The first step scales the tile brightness by a certain value (e.g., from -1.0 to 1.0) based on the signal KNEE. The AI ​​metering control module 320 can iterate through the brightness values ​​of all tiles in the 3A statistics. If the 3A statistics indicate that the tile brightness is less than 2048, the value can be scaled by 1 + KNEE. If the 3A statistics indicate that the tile brightness is greater than or equal to 2048, the value can be scaled by 1 - KNEE. The second step may be to calculate the weighted average brightness. After determining the tile brightness and tile weights, the weighted tile brightness can be calculated using equation EQ1:

[0157] EQ1: The final equation for weighted tile brightness = tile brightness × tile weight (e.g., for CLUMA signal communication) can be calculated using EQ2. The value N in equation EQ2 can be the maximum number of tiles.

[0158] EQ2:

[0159] If the signal LROI indicates that there is no current computer vision result in the current video frame among video frames 302a-302n, but there are some historical computer vision results in the last tracking loop number of the video frames, then the AI ​​metering control module 320 can operate in an operating mode that provides a metering strategy using previous luminance values. The AI ​​metering control module 320 can use the same luminance value from the previous tracking loop of video frames 302a-302n as the current luminance value CLUMA.

[0160] When there is no current computer vision result but some historical information does exist, the AE control for the current frame in frames 302a-302n may not change. For example, the AI ​​metrology control module 320 may operate under the assumption that the result of the computer vision operation performed by the ROI detection module 306 is delayed (e.g., the computer vision result may still be processing data), and the data may be expected to arrive in the next tracking loop number of video frames 302a-302n. If no computer vision result is still received, the AI ​​metrology control module 320 may change its operating mode to provide a metrology strategy for the entire FOV (e.g., as if there is no history). In the example, when there is no current computer vision result but some previous history exists in the tracking loop, the scene may be that the object / subject has moved out of the FOV, and the computer vision operation no longer detects any subject of interest (e.g., the object / subject subsequently returns and the same calculation may be appropriate).

[0161] The tracking loop number signal TLN can be a user input to the AI ​​metering control module 320. The tracking loop number set by the TLN signal can be a threshold for the number of video frames, configured to indicate when the AI ​​metering control module 320 can switch from ROI metering operation mode to full FOV metering mode. The tracking loop number can indicate a unit of time equal to the frame time of video frames 302a-302n. In the example, if the TLN signal provides a threshold of 10, then if the AI ​​metering control module 320 does not receive computer vision results from the ROI detection module 306 in the last ten consecutive video frames 302a-302n, the AI ​​metering control module 320 can switch from the ROI metering operation mode based on historical information (e.g., a historical list of ROIs stored in the tracking list queue 324) to the full FOV metering operation mode.

[0162] Using the TLN signal to set the tracking cycle number prevents constant switching of operating modes. For example, a very low tracking cycle number (e.g., a value of 1) could cause a change to full FOV metering operating mode at any time, in the presence of any delay in computer vision results or any communication failure. Switching between operating modes (e.g., frequent changes between ROI metering using historical data and full FOV metering mode) can lead to unstable AE control (e.g., flickering due to constantly changing AE parameters selected with different operating modes). Typically, the TLN signal can be selected as a tracking cycle number value that is longer than the amount of time the ROI detection module 306 performs computer vision on one of the video frames 302a-302n.

[0163] In the example, the AI ​​metering control module 320 can receive 3A statistics corresponding to the frame number of one of the video frames 302a-302n. If the AI ​​metering control module 320 does not receive a computer vision result for a specific video frame, it can check the tracking list queue 324 for any historical information. The AI ​​metering control module 320 can receive a signal HTS and compare any of the computer vision results stored in the tracking list queue 324 to see if it is within the last tracking loop number of the video frame. If one of the computer vision results stored in the tracking list queue has a historical timestamp HTS within the last tracking loop number of the video frame, the AI ​​metering control module 320 can use historical data (e.g., a previous current luminance value can be represented as a signal CLUMA) to select the ROI tracking operation mode. If none of the historical data has a historical timestamp HTS within the last tracking loop number of the video frame, the AI ​​metering control module 320 can select the full FOV metering operation mode.

[0164] If the signal LROI indicates the presence of a current computer vision result in the current video frame (302a-302n), the AI ​​metrology control module 320 can operate in an operating mode that provides a metrology strategy using ROI tracking. Calculation of the current luminance value CLUMA can prioritize the ROI area indicated by the signal LROI. The current luminance value CLUMA can be calculated to ensure optimal exposure for the ROI area.

[0165] The AI ​​metering control module 320 can analyze information about each ROI in the ROI list provided in the signal LROI. The AI ​​metering control module 320 can iterate over the ROI information, repeating the same operation for each group of ROI information in the ROI list. After performing the operation on a group of ROI information, the AI ​​metering control module 320 can analyze the next group of ROI information. When the end of the ROI list is reached, the AI ​​metering control module 320 can exit the loop and then perform a weighted average calculation using the tile brightness values ​​in the signal 3A_STAT and the evaluated ROI information.

[0166] The AI ​​measurement control module 320 can perform ROI prediction. ROI prediction can be performed in response to the difference between the timestamp in signal TS and the updated timestamp in signal TS'. Since the computation time of the computer vision operation performed by the ROI detection module 306 may vary, the time when the signal LROI with the computer vision result is reported to the AI ​​measurement control module 320 may not necessarily be real-time. The updated timestamp TS' for the current computer vision result may differ from the timestamp TS corresponding to the current 3A statistics in signal 3A_STAT. If the timestamp difference (e.g., TS-TS') is zero, the AI ​​measurement control module 320 can skip ROI prediction. If the timestamp difference is not zero, the AI ​​measurement control module 320 can perform ROI prediction.

[0167] The AI ​​measurement control module 320 can determine ROI predictions based on historical ROI information stored in the tracking list queue 324. In the example, the AI ​​measurement control module 320 can be configured to compute one or more fitting functions based on historical points. After applying the current time (e.g., timestamp TS) to the fitting function, the AI ​​measurement control module 320 can perform ROI predictions by estimating the real-time ROI location.

[0168] After calculating the real-time ROI location, the AI ​​metering and control module 320 can compare the coordinates of the real-time ROI with the coordinates of each tile in one of the video frames 302a-302n. For example, the sub-regions / tiles of video frames 302a-302n can be classified as ROI tiles, cross tiles, and / or background tiles. This can be combined with... Figure 7 Describe the details of the classification of various tiles / sub-regions in video frames 302a-302n.

[0169] After all tiles are categorized, the AI ​​metering control module 320 can calculate the average brightness value for each tile category. If the ROI is too small to encompass a single tile, the AI ​​metering control module 320 can skip calculating the average brightness value for the small ROI (e.g., brightness values ​​for the small ROI may not be generated). If none of the ROIs have valid ROI tiles, the AI ​​metering control module 320 can consider the current computer vision result invalid. If invalid computer vision results exist, the AI ​​metering control module 320 can change the operating mode to full FOV calculation (e.g., no computer vision results without history) or historical value operating mode.

[0170] In response to processing all ROI information in the ROI list, the AI ​​metering control module 320 can calculate a weighted average brightness. The weighted average brightness calculation can be determined based on the average brightness of the ROI tiles, the average brightness of the background tiles, and the ROI weighting. The weighted average brightness value calculation can be used to generate a signal CLUMA that includes the weighted average brightness on both the ROI tiles and the background tiles. Other factors can also be used to determine the signal CLUMA. In this example, the accuracy of the weighted average brightness value calculation may depend on the movement speed of the ROIs in the ROI list. Generally, the faster the ROI moves, the less accurate the calculated weighted average brightness value may be. The AI ​​metering control module 320 can also generate the signal CLUMA in response to information in the tracking list queue 324, the signal 3A_STAT, and the signal LROI, to generate the signal CLUMA based on the weighted average brightness value and the original brightness value. The signal CLUMA, including the weighted average brightness and / or the original brightness value, can be provided to the AE control module 322. After generating the signal CLUMA, the AE control module 322 can update the historical computer vision results in the tracking list queue 324. In the example, the tracking list queue 324 can maintain the latest three computer vision results.

[0171] refer to Figure 6 The diagram illustrates object detection and tracking in an example video frame before the application of AI exposure metering. Example video frame 350 is shown. Example video frame 350 can be a YUV image provided to ROI detection module 306 in the signal YUV. Example video frame 350 can be an illustrative example of a video frame in a sequence of video frames 302a-302n analyzed / evaluated by processor 102. In the illustrated example, example video frame 350 can correspond to a time when computer vision ROI detection and / or analysis has been performed by ROI detection module 306 but AI metering has not yet been processed by AI metering control module 320.

[0172] Example video frame 350 may include pixel data of a video frame arranged to be provided to CNN module 190b. CNN module 190b may be configured to analyze the pixel data of video frame 350 to detect the size and / or location of various types of objects / subjects captured in video frame 350. In the example, a computer vision AI model implemented by ROI detection module 306 may be configured to detect various objects in example video frame 350. Example video frame 350 may include an RGB image.

[0173] Example video frame 350 may be a video frame captured in an outdoor area (e.g., a view from camera system 100 implementing capture device 104). In one example, example video frame 350 may be a video frame captured outdoors by a smartphone camera. In another example, example video frame 350 may be video captured by a smart doorbell and / or home security camera. In the example shown, example video frame 350 may be video captured by a camera configured to focus on detecting pedestrians. The use cases for capturing and analyzing example video frame 350 may vary depending on the design criteria of a particular implementation.

[0174] Example video frame 350 may include a sun 352, multiple background objects 356a-356n, and people 362a-362b. The sun 352 may be emitting light that can be captured by sensor 180. Sunlight may cause areas of example video frame 350 to be very bright or cast shadows on some areas of example video frame 350. Background objects 356a may be buildings, and background objects 356b-356n may be trees. Typically, background objects 356a-356n may not be objects of interest for ROIs in the ROI list that can be detected by the computer vision operation of ROI detection module 306. People 362a-362b may be objects of interest for ROIs in the ROI list that can be detected by the computer vision operation of ROI detection module 306. Person 362a is shown to be closer to image sensor 180 than person 362b, and therefore appears larger. Person 362a may be walking towards lens 160. Person 362b is shown further away from image sensor 180 than person 362a and may appear smaller. Person 362b is shown walking within the field of view of lens 160 (e.g., not moving closer to or further away from lens 160).

[0175] Dashed shapes 370a-370b are shown. Dashed shapes 370a-370b can represent the detection of objects / subjects by computer vision operations performed by processor 102. Dashed shapes 370a-370b can each include pixel data corresponding to objects detected by the computer vision operation pipeline and / or CNN module 190b. In the illustrated example, dashed shapes 370a-370b can be detected in response to pedestrian detection operations performed by the computer vision operation pipeline and / or CNN module 190b (e.g., a computer vision AI model implemented by ROI detection module 306). Dashed shapes 370a-370b are shown for illustrative purposes. In the example, dashed shapes 370a-370b can be a visual representation of object detection (e.g., dashed shapes 370a-370b may not appear on the output video frame). In another example, the dashed shapes 370a-370b can be bounding boxes generated by processor 102, which are displayed on the output video frame to indicate that an object has been detected (e.g., bounding boxes 370a-370b can be displayed in debug operation mode).

[0176] Computer vision operations can be configured to detect characteristics of detected objects, behaviors of detected objects, directions of movement of detected objects, and / or the presence of detected objects. Object characteristics may include height, length, width, slope, arc length, color, emitted light intensity, detected text on the object, movement path, movement speed, direction of movement, proximity to other objects, etc. Detected object characteristics may include the object's state (e.g., open, closed, on, off, etc.). Behavior and / or presence can be determined in response to the object's type and / or the characteristics of the detected objects. Although an example video frame 350 is shown, object behavior, direction of movement, and / or presence can be determined by analyzing a sequence of video frames 302a-302n captured over time. For example, movement path and / or speed characteristics can be used to determine whether an object classified as a person is walking or running. The type of detected characteristics and / or behaviors can vary depending on the design criteria of a particular implementation.

[0177] In the illustrated example, bounding box 370a may be the region of interest for a large person 362a, and bounding box 370b may be the region of interest for a small person 362b. In the example, the settings (e.g., feature set) of processor 102 (e.g., a computer vision AI model implemented by ROI detection module 306) may define the object of interest as a person and / or face. For example, trees, plants, decorations, furniture, vehicles, and / or buildings may not be objects of interest for the feature set defined for detecting people and / or faces. In the illustrated example, bounding boxes 370a-370b are shown as having a square (or rectangular) shape. In some embodiments, the shape of the bounding boxes 370a-370b corresponding to detected pedestrians may be formed to follow the body shape and / or facial shape of people 362a-362b (e.g., an irregular shape following the curves of the facial and / or body shape of people 362a-362b).

[0178] Processor 102 and / or CNN module 190b can be configured to implement pedestrian and / or face detection techniques. In some embodiments, processor 102 and / or CNN module 190b can detect other types of subjects as objects of interest (e.g., animals, vehicles, moving objects, fallen objects, etc.). Computer vision techniques can be performed on the YUV image provided by IDSP 304 to ROI detection module 306.

[0179] Computer vision techniques can be configured to detect Regions of Interest (ROIs) and / or generate information about ROIs corresponding to subjects 362a-362b. For example, bounding boxes 370a-370b can be visual representations of the detected ROIs. Processor 102 can loop through computer vision techniques (e.g., iteratively performing object / subject detection throughout example video frame 350) to determine whether any object of interest (e.g., as defined by a feature set) exists within the field of view of lens 160 and / or image sensor 180. In the illustrated example, by iteratively performing ROI detection techniques, processor 102 can detect pedestrians 362a within ROI 370a and 362b within ROI 370b.

[0180] Although only pedestrians 362a-362b are shown as ROIs 370a-370b, computer vision operations performed by processor 102 and / or CNN module 190b can be configured to detect background objects 356a-356n. Background objects 356a-356n can be detected for other computer vision purposes (e.g., training data, labeling, depth detection, etc.), but may not be used for the purposes of AI econometric techniques performed by device 100. The subjects(s) of the ROIs may not be limited to human detection, as shown in example video frame 350. The types(s) of subjects(s) identified for ROIs 370a-370b can vary depending on the design criteria of a particular implementation.

[0181] Arrows MDIR[A] and MDIR[B] are shown. Arrow MDIR[A] may extend in the direction from ROI 370a. Arrow MDIR[B] may extend in the direction from ROI 370b. Arrows MDIR[A]-MDIR[B] can provide a visual representation of the direction of movement of objects (e.g., pedestrians 362a-362b) within ROIs 370a-370b. The movement arrows MDIR[A]-MDIR[B] are shown for illustrative purposes. In the example, the movement arrows MDIR[A]-MDIR[B] may be a visual representation of movement of the ROI (e.g., the movement arrows MDIR[A]-MDIR[B] may not appear on the output video frame). In another example, the movement arrows MDIR[A]-MDIR[B] can be visual indicators generated by processor 102 and displayed on the output video frame to indicate that the movement of ROI 370a-370b has been detected and / or tracked (e.g., the movement arrows MDIR[A]-MDIR[B] can be displayed in debug operation mode).

[0182] The movement arrow MDIR[A] can indicate that person 362a can move in a direction toward lens 160. For example, in the sequence of video frames 302a-302n, person 362a can become larger and occupy more tiles in video frames 302a-302n (e.g., when person 362a moves closer). The movement arrow MDIR[B] can indicate that person 362b can move in a direction from left to right across lens 160. For example, in the sequence of video frames 302a-302n, person 362b can remain the same size and occupy the same number of tiles in video frames 302a-302n, but the number of tiles occupied can change as person 362b moves.

[0183] Processor 102 (e.g., CNN module 190b implementing ROI detection module 306) can determine whether any object / subject exists in the analyzed video frame. In example video frame 350, the object of interest could be pedestrians 362a-362b. If pedestrians have been captured in video frame 350, ROI detection module 306 can record the size and offset of the object (e.g., its location relative to the edge of video frame 350) to define ROIs 370a-370b. ROI detection module 306 can also generate ROI information for ROIs 370a-370b. In the example shown, processor 102 can detect pedestrians 362a-362b. Processor 102 can store information about ROIs 370a-370b in memory 150 and / or present a signal LROI including ROI information to 3A module 308.

[0184] ROI detection module 306 can detect ROIs 370a-370b. In the example shown, the ROI list may include ROI 370a and ROI 370b (e.g., two entries). ROI information 380a-384b is shown for ROIs 370a-370b. The ROI information may include ROI IDs 380a-380b, ROI weights 382a-382b, and ROI locations 384a-384b, each corresponding to ROIs 370a-370b respectively.

[0185] ROI IDs 380a-380b can be unique values ​​used to distinguish each of ROIs 370a-370b. For example, ROI IDs 380a-380b can be integer values. In the example shown, ROI ID 380a can be zero for ROI 370a, while ROI ID 380b can be one for ROI 370b (e.g., any additional objects detected can be identified by the next incrementing number). Other forms of identification can be used (e.g., hash values, letter values, user-provided names, etc.).

[0186] The ROI weights 382a-382b can indicate which of the ROIs 370a-370b has a higher exposure priority. Computer vision operations can determine that person 362a may have a higher exposure priority than person 362b (e.g., because person 362a is closer to the capture device 104 and may occupy more space in video frame 350). The ROI detection module 306 can determine that both people 362a-362b should be well exposed, but person 362a has a higher priority. In the example shown, the ROI weight 382a could be 0.7 for ROI 370a, while the ROI weight 382b could be 0.3 for ROI 370b (e.g., totaling 1.0, where ROI 370a has more weight). In some embodiments, if AI metrics have not yet been applied, people 362a-362b, the sun 352, and background objects 356a-356n (e.g., buildings, trees, etc.) can be weighted equally, and the entire FOV of example video frame 350 can have the same weighting value.

[0187] ROI locations 384a-384b may include the x, y coordinates of detected ROIs 370a-370b. In the example, ROI location 384a of ROI 370a may include x, y coordinates (or offsets) indicating that ROI 370a is located on the right and occupies most of the right side of video frame 350. In another example, ROI location 384b of ROI 370b may include x, y coordinates (or offsets) indicating that ROI 370b is located near the top left corner and occupies a relatively small portion of video frame 350.

[0188] The ROI list and / or ROI information can be presented to the 3A module 308 via the signal LROI. The ROI list and / or ROI information can be stored in the tracking list queue 324. The example video frame 350 can be an example video frame captured without applying AI metering. The ROI list and / or ROI information can be used to implement AI metering for upcoming video frames.

[0189] The trace list indicator 386 is shown. The trace list indicator 386 can include a "false" value. A false value for the trace list indicator 386 can indicate that there may not be any historical information currently in the trace list queue 324. For example, example video frame 350 could be the first in a sequence of video frames 302a-302n.

[0190] In example video frame 350, without AI metrology applied, the sun 352 is shown as slightly overexposed, while background objects 356a-356n may be slightly dark. Pedestrians 362a-362b detected by computer vision AI may be very dark. For example, pedestrians 362a-362b are shown as having shadows to indicate poor exposure in example video frame 350 without AI metrology applied. In one example, generating video frames 302a-302n with very dark key objects (e.g., pedestrians 362a-362b) may prevent further analysis (e.g., post-processing, face identification, etc.). In some embodiments, each of pedestrians 362a-362b may be included in metrology calculations performed by AI metrology control module 320. In some embodiments, computer vision operations performed by ROI detection module 306 may be applied only to specific (e.g., people of interest) and exclude others. For example, people of interest may be predefined, familiar people that can be stored in memory 150. In one example, pedestrian 362a could be a person of interest who can be used as an ROI for AI metering techniques, while pedestrian 362b might not be a person of interest and might not be included in the list of ROIs for AI metering techniques. AI metering techniques can be configured to correct exposure and / or provide consistent exposure for subjects defined by ROIs 370a-370b.

[0191] refer to Figure 7 A diagram illustrating the region of interest tile classification of an example video frame is shown. Example video frame 400 is shown. Example video frame 400 can be a representation of example video frames 302a-302n. In the example shown, video frame 400 can be... Figure 6 A version of example video frame 350 is shown in association, illustrating tiles / sub-regions that can be classified by the AI ​​metering control module 320. In the example shown, example video frame 400 may correspond to the time when the AI ​​metering control module 320 is performing ROI metering control.

[0192] The AI ​​metering control module 320 can receive a signal LROI that includes a list of ROIs and ROI information. ROIs 370a-370b are shown in example video frame 400. In the example shown, ROIs 370a-370b can correspond to... Figure 6 The same location is shown in the example video frame 350, which is illustrated in association. The AI ​​metering control module 320 can analyze ROI information about ROIs 370a-370b in the ROI list to determine the current brightness value. Since the ROI list can include two ROIs 370a-370b, the AI ​​metering control module 320 can apply AI metering technology using an ROI tracking operation mode to calculate the current brightness value.

[0193] The AI ​​metering control module 320 can perform analysis and / or calculations based on ROI information in the ROI list and / or tile brightness values ​​in 3A statistics. Because analysis can be performed on the ROI information, analysis and / or calculations can be performed regardless of the type of the detected object / subject. For example, the ROI detection module 306 can implement various computer vision AI modules to detect ROI information. As long as ROI information is provided, the AI ​​metering control module 320 can perform the same analysis and / or calculations (e.g., the type of object may not matter).

[0194] Example video frame 400 is shown as being overlaid with a grid pattern. The grid pattern may include multiple tiles / sub-regions 402aa-402nn. Each of the tiles 402aa-402nn may have a tile brightness value provided by the signal 3A_STAT generated by IDS 304. In the example shown, the first row of the grid pattern may include tiles 402aa-402an, the second row of the grid pattern may include tiles 402ba-402bn, the last row of the grid pattern may include tiles 402na-402nn, and so on. The grid pattern may include n×n tiles. The number of rows and columns of the grid pattern for tiles 402aa-402nn may be different values ​​(e.g., m×n grid). In the example shown, the grid pattern for tiles 402aa-402nn may be 16×16 granularity. In other embodiments, the granularity of the grid pattern for tiles 402aa-402nn may be 12×8, 24×16, 32×32, etc. The granularity of the grid pattern for tiles 402aa-402nn can vary according to the design criteria for a specific implementation.

[0195] The AI ​​metering control module 320 can be configured to classify each of the tiles 402aa-402nn. Tiles 402aa-402nn can be classified as ROI tiles, intersection tiles, or background tiles. Any tile within one of the ROIs 370a-370b in tiles 402aa-402nn can be an ROI tile. Any tile within tiles 402aa-402nn that includes the boundary of ROIs 370a-370b (e.g., intersects with a bounding box) can be an intersection tile. Any tile outside of ROIs 370a-370n in tiles 402aa-402nn can be a background tile.

[0196] ROI 370a may include shadow tiles 410aa-410am and crosshair tiles 412aa-412am. Shadow tiles 410aa-410am may be crosshair tiles of ROI 370a. Crosshair tiles 412aa-412am may be ROI tiles of ROI 370a. ROI 370b may include shadow tiles 410ba-410bn and crosshair tiles 412ba-412bn. Shadow tiles 410ba-410bn may be crosshair tiles of ROI 370b. Crosshair tiles 412ba-412bn may be ROI tiles of ROI 370b. Other tiles in tiles 402aa-402nn that do not have shadows or crosshairs may be background tiles. In the example shown, ROI 370a may include 20 ROI tiles 412aa-412am (e.g., 10x2) and 28 cross tiles 410aa-410am, and ROI 370b may include 6 ROI tiles 412ba-412bn (e.g., 6x1) and 18 cross tiles 410ba-410bn. The number of cross tiles or ROI tiles in each of ROIs 370a-370b may vary depending on the size of ROIs 370a-370b.

[0197] Each tile in tiles 402aa-402nn can correspond to a sub-region of the YUV image. Each tile in tiles 402aa-402nn can include a brightness value for a specific sub-region. The brightness value quantifies the brightness of the sub-region. For ROI 370a, ROI tiles 412aa-412am can be completely enclosed by the bounding box of ROI 370a. Similarly, for ROI 370b, ROI tiles 412ba-412bn can be completely enclosed by the bounding box of ROI 370b.

[0198] For ROI 370a, the bounding box can cross tiles 410aa-410am. Similarly, for ROI 370b, the bounding box can cross tiles 410ba-410bn. Since tiles 410aa-410am and / or 410ba-410bn may include a portion of the ROI brightness and a portion of the background brightness, the brightness values ​​of the cross tiles may be inaccurate for reference. The AI ​​metering control module 320 may not use tiles 402aa-402nn that are classified as cross tiles in the current brightness value calculation. If any ROI in ROIs 370a-370b does not include any of the ROI tiles but only has cross tiles, the computer vision result for that ROI may be considered an invalid computer vision result. Invalid computer vision results may not be stored in the tracking list queue 324 as part of the historical computer vision results.

[0199] The remaining tiles in (shadowless) tiles 402aa-402nn can be background tiles. Background tiles may not intersect with or be surrounded by ROI 370a or ROI 370b. Background tiles can represent the brightness values ​​of the background of a YUV image.

[0200] ROI tiles and background tiles can be weighted based on the current brightness value. The average brightness value of the ROI tiles for each ROI in ROIs 370a-370n can be calculated according to Equation 3.

[0201] EQ3: The average brightness value of the background tiles can be calculated using equation EQ4.

[0202] EQ4: The AI ​​metering and control module 320 can use the average brightness value of each ROI (ROI). Li ), and the average brightness of the background (BG_L) and the ROI weighted (ROI) Wi The current brightness value can be calculated using equation EQ5.

[0203] EQ5: The value N can be the number of ROIs in the ROI list. For example, for video frame 400, the value of N could be 2 (e.g., ROIs 370a-370b). Other video frames in video frames 302a-302n can include different amounts of ROIs.

[0204] The current luminance value can be calculated by the AI ​​metering control module 320 according to equation EQ5. The calculated current luminance value can be presented to the AE control module 322 via the signal CLUMA. The AE control module 322 can be configured to compare the current luminance value with the target luminance value to determine the AE parameters. The AE control module 322 can provide the updated AE parameters to the capture device 104 via the signal AE_PARAM. The signal AE_PARAM enables video frames 302a-302n captured after the example video frame 350 to be generated with consistent exposure for the detected object / subject.

[0205] refer to Figure 8 The diagram illustrates object detection and tracking in an example video frame after applying AI exposure metering. An example adjusted video frame 450 is shown. The adjusted video frame 450 can be included in... Figure 6One of the video frames 302a-302n generated after the example video frame 350 shown in association. For example, the example video frame 350 may have been generated and analyzed / evaluated by the dynamic AI metrology component 300, the AI ​​metrology technology may have calculated AE parameters in response to the example video frame 350, and the adjusted video frame 450 may have been captured after the capture device 104 has been adjusted according to the calculated AE parameters. In the example shown, the adjusted video frame 450 may correspond to the time when the AI ​​ROI metrology control has taken effect and the AE has stabilized.

[0206] Since the adjusted video frame 450 may have been captured shortly after the example video frame 350, it may include similar content. The adjusted video frame 450 may include the sun 352, background objects 356a-356n, and pedestrians 362a-362b. In the adjusted video frame 450, pedestrians 362a-362b may have moved compared to their positions in the example video frame 350. Movement arrows MDIR[A]-MDIR[B] are shown to indicate the direction of movement of the corresponding pedestrians 362a-362b. ROIs 370a-370b corresponding to pedestrians 362a-362b are shown. Background objects 356a-356n may be static and have not moved between video frames.

[0207] A flare effect 452 is shown. The flare effect 452 may be located around the sun 352. A pedestrian 362c is shown. Pedestrian 362c may have entered the field of view of image sensor 180 after example video frame 350 (e.g., after AE parameters have been calculated). A dashed box 454 is shown around pedestrian 362c. The dashed box 454 may represent the bounding box of the ROI detected for pedestrian 362c. The ROI 454 for pedestrian 362c may be generated similarly to ROIs 370a-370b. An arrow (e.g., MDIR[C]) is shown. The arrow MDIR[C] may include a movement arrow indicating the direction of movement being tracked for pedestrian 362c.

[0208] In the adjusted video frame 450, ROIs 370a-370b and the new ROI 454 can have ROI information. ROI 370a can have the same ROI ID 380a with a value of zero, ROI 370b can have the same ROI ID 380b with a value of one, and the new ROI 454 can have the next value of ROI ID 380c with a value of two.

[0209] In response to the new location / size of detected objects and / or any new or missing objects, the ROI detection module 306 can update the ROI weights 382a-382c while performing computer vision operations. In the example shown, the ROI 370a with the largest size may have the highest ROI weight 382a of 0.6, the second largest ROI 370b may have the second highest ROI weight 382b of 0.3, and the new ROI 454 (e.g., not previously in the tracking list queue 324) may have the lowest ROI weight 382c of 0.1. Typically, ROIs previously in the tracking list queue 324 may have a higher weight than the new ROI. The ROI weights 382a-382b may have been rebalanced to include the new ROI 454 while still providing a total weight of 1.0.

[0210] In the illustrated example, the ROI position 384a of ROI 370a may include x, y coordinates (or offsets) indicating that ROI 370a is located in the lower right and occupies a relatively large portion of the adjusted video frame 450 compared to the previous example video frame 350. In the illustrated example, the ROI position 384b of ROI 370b may include x, y coordinates (or offsets) indicating that ROI 370b is located in the middle and occupies a similar portion of the adjusted video frame 450 compared to the previous example video frame 350. In the illustrated example, the ROI position 384c of the new ROI 454 may include x, y coordinates (or offsets) indicating that ROI 454 is located in the upper right and occupies a relatively small portion of the adjusted video frame 450.

[0211] The adjusted video frame 450 can be an example video frame captured with AI metering applied. A tracking list indicator 386 is shown. The tracking list indicator 386 can include a "true" value. A true value for the tracking list indicator 386 can indicate that historical information may currently exist in the tracking list queue 324. For example, historical information used for the example video frame 350 can be stored in the tracking list queue 324.

[0212] After applying AI metrology, the exposure of the adjusted video frame 450 may be consistent with that of the example video frame 350. In the adjusted video frame 450, the sun 352 may be highly overexposed. The flare effect 452 can provide an illustrative representation of the overexposure. In the adjusted video frame 450, the background objects 356a-356n may be slightly overexposed (e.g., compared to the slightly darker appearance in the example video frame 350).

[0213] After AI metrology has been applied, the object / subject (e.g., pedestrians 362a-362b) can be well exposed. In the example shown, pedestrians 362a-362b are shown without shadows to indicate consistent exposure (e.g., compared to the shadows shown in example video frame 350). Since pedestrians 362a-362b are well exposed in the adjusted video frame 450, the video data corresponding to pedestrians 362a-362b may be suitable for further analysis purposes (e.g., face identification, post-processing, depth analysis, etc.). AI metrology can generate well-exposed objects even in more challenging scenarios. For example, if one of the pedestrians 362a-362b is under a roof (e.g., in shadow), there may be a larger exposure ratio between the ROI of pedestrians 362a-362b and the background. Even with a large exposure ratio, AI metrology can be configured to generate AE parameters to achieve good exposure of the object of interest.

[0214] In the adjusted video frame 450, the new pedestrian 362c is shown as having shadows. Since the new pedestrian 362c may have entered the field of view of the image sensor 180 after the AE parameters were generated, it may not yet be well exposed. In the example shown, the new pedestrian 362c is shown as dark, but it could alternatively be overexposed depending on the previous AE parameters for a particular location. For example, the new pedestrian 362c may not be well exposed before the ROI detection module 306 analyzes information about ROI 454 and the AI ​​metering control module 320 analyzes the updated ROI list and recalculates the current brightness value based on the updated ROI information. In the adjusted video frame 450, the new pedestrian 362c may have a different brightness than the other two pedestrians 362a-362b (e.g., which have already been included in the AI ​​metering area).

[0215] After analyzing the adjusted video frame 450, the AI ​​metering calculation can be made to include the new pedestrian 362c. The weighted average brightness value can be changed when calculating the current brightness value for the adjusted video frame 450. The AE parameters can be updated by the AE control module 322, and the next video frame in video frames 302a-302n can be further adjusted, as can the exposure of all pedestrians 362a-362c. For example, the exposure for previously metered pedestrians 362a-362b can be slightly adjusted, while a larger adjustment can be made to the exposure for the new pedestrian 362c to ensure consistent exposure for all objects of interest.

[0216] The computation time of a computer vision operation can vary depending on how the operation is implemented. Because the time series may change, the updated timestamp TS' used for the ROI list may differ from the timestamp TS used for the 3A statistics. If the two timestamps differ, the computer vision results may be out of sync with the current 3A statistics. The AI ​​measurement control module 320 can be configured to perform predictions on the current ROI information to compensate for the lack of synchronized data.

[0217] The prediction operation can begin at the starting point of the ROI, and the prediction for the ending point of the ROI can follow the same process. To perform the prediction operation, the AI ​​measurement control module 320 can rely on historical ROI data. For example, the tracking list queue 324 can store three previous computer vision results. Depending on the availability of historical data, the prediction operation may include different steps.

[0218] If the ROI ID is new (e.g., ROI 454 for the new pedestrian 362c), there may be no historical data. When there is no historical data, prediction can be skipped. The ROI information at the updated timestamp TS' can be used to calculate the brightness for the ROI at timestamp TS. Since there may be a mismatch between the updated timestamp TS' and the timestamp TS, the brightness calculation for the new ROI 454 may include some inaccuracies. However, as long as the new pedestrian 362c does not move fast enough to move out of the previous ROI within the time difference (e.g., TS-TS'), the real-time location of the new pedestrian 362c may still have some overlapping ROI locations from the updated timestamp TS'. The calculated brightness values ​​can be accurate to some extent, so the brightness results will be used. When no historical data is available, the accuracy of the prediction may depend on the movement speed of (multiple) objects. Generally, even if the movement speed is high enough that the ROI prediction is inaccurate, the prediction used can still provide more consistent exposure than without using AI metering.

[0219] When an ROI ID (e.g., ROI 370a) with historical information about the ROI location exists, and a new current ROI location exists for ROI 370a, prediction can be configured to compute a linear fitting function with respect to x, y (the x and y coordinates of the ROI's starting point) and time t. For example, ROI 370a might have coordinates x1, y1 at timet1, and the new location coordinates for ROI 370a might have coordinates x2, y2 at timet2. The AI ​​measurement control module 320 can compute the fitting function based on the known values ​​of x1, x2, t1, and t2. The coefficients a and b in the first function EQ6 can be computed using two sets of data (x1, t1) and (x2, t2). The coefficients c and d in the second function EQ7 can be computed using (y1, t1) and (y2, t2).

[0220] EQ6: x = a × t + b

[0221] EQ7: y = c × t + d

[0222] After the AI ​​measurement and control module 320 generates the fitting function, the current time TS can be applied to the fitting function to predict the starting point of the ROI.

[0223] When the same ROI ID (e.g., ROI 370a) has two historical data points (e.g., ROI locations) at time T1 and time T2, and a new ROI location at time T3, the prediction operation can compute a quadratic fitting function for the current x, y coordinates and time t. The coefficients a, b, and c of the first function EQ8 can be computed using three known sets of data (x1, t1), (x2, t2), and (x3, t3). The coefficients d, e, and f of the second function EQ9 can be computed using (y1, t1), (y2, t2), and (y3, t3).

[0224] EQ8: x = a × t 2 +b×t+c

[0225] EQ9: y = d × t 2 +e×t+f

[0226] After the AI ​​measurement control module 320 generates the fitting function, the current time TS can be applied to the fitting function to predict the starting point of ROI 370a.

[0227] For scenarios with more than two historical data points, the prediction operation can be configured to predict ROI locations based on a cubic fitting function. However, the computation time and memory storage costs may increase. Using cubic fitting can involve trade-offs between time, space, and accuracy. Whether the AI ​​measurement control module 320 uses a cubic fitting function can depend on user preference. For example, the user can use HID 166 to provide the signal USER to enable / disable cubic fitting. Typically, if a quadratic fitting function provides sufficient accuracy, the system can use the two most recent historical data points and new data using equations EQ8 and EQ9 to predict points on the quadratic fitting function.

[0228] refer to Figure 9 The diagram illustrates method (or process) 500. Method 500 enables automatic exposure metering by tracking the region of interest of a moving object using artificial intelligence. Method 500 generally includes steps (or states) 502, 504, 506, 508, 510, 512, 514, 516, a decision step (or state) 518, 520, and 522.

[0229] Step 502 can initiate method 500. In step 504, processor 102 can receive pixel data. In this example, capture device 104 can generate pixel data in response to light input signal LIN and generate a signal VIDEO including the pixel data. Next, in step 506, processor 102 can perform video processing operations to process the pixel data arranged as video frames 302a-302n. In step 508, IDSP 304 of processor 102 can be configured to generate statistics and timestamps for the next video frame in video frames 302a-302n. In this example, statistics may include 3A statistics, which can be presented to 3A module 308 via signal 3A_STAT, and timestamps may be timestamps corresponding to one of the video frames in video frames 302a-302n and can be communicated to 3A module 308 and ROI detection module 306. Next, method 500 can proceed to step 510.

[0230] In step 510, processor 102 can perform computer vision operations to determine the regions of interest (ROIs) at timestamps in the analyzed video frames. In this example, ROI detection module 306 (e.g., a component of CNN module 190b) can detect ROIs corresponding to desired objects / events and generate a list of ROIs (LROIs) at times corresponding to the updated timestamp signal TS'. Next, in step 512, ROI detection module 306 and 3A statistics module 310 can track the movement of ROIs across the sequence of video frames 302a-302n. For example, when ROI detection module 306 detects an ROI and updates the ROI positions 384a-384c corresponding to the respective ROI IDs 380a-380c, historical ROI information can be stored in the tracking list queue 324 via historical timestamps (HTS) and the historical ROI information list (HLROI). In step 514, processor 102 can evaluate the current brightness value of the ROI. In the example, the AI ​​metering control module 320 can receive ROI information from the signal LROI, the knee value from the signal KNEE, the metering weighting table from the signal MWT, the historical ROI list information from the signal HLROI, and the 3A statistics from the signal 3A_STAT to evaluate the current luminance value. Next, in step 516, the processor 102 can compare the current luminance value with the target luminance value. In the example, the AE control module 322 can compare the current luminance value in the signal CLUMA with the target luminance value in the signal TLUMA. Next, method 500 can move to decision step 518.

[0231] In decision step 518, processor 102 can determine whether to update the AE parameters. In the example, the AE parameters can be updated to adjust the current brightness value to match the target brightness value. For example, updating the AE parameters may not be beneficial if the current brightness value is already at the target brightness value. If it is determined that there is no need to update the AE parameters, method 500 can return to step 504 (e.g., to evaluate the next video frame in video frames 302a-302n). If it is determined that there is a need to update the AE parameters, method 500 can move to step 520. In step 520, processor 102 can calculate the AE parameters in response to the current brightness value and the target brightness value. For example, the AE parameters can be calculated to align the current brightness value with the target brightness value. Then, in step 522, processor 102 can generate the AE parameters. For example, AE control module 322 can present the signal AE_PARAM to capture device 104 (e.g., to adjust DC aperture, shutter speed, AGC, etc.). Next, method 500 can return to step 504 (e.g., to evaluate the next video frame in video frames 302a-302n).

[0232] refer to Figure 10 The diagram illustrates method (or process) 550. Method 550 may select an automatic exposure metering technique. Method 550 generally includes steps (or states) 552, 554, decision steps (or states) 556, 558, decision steps (or states) 560, 562, 564, 566, 568, and 570.

[0233] Step 552 can initiate method 550. In step 554, the ROI detection module 306 can generate ROI information. For example, if a computer vision result is detected, the ROI detection module 306 can generate a signal LROI to update the ROI information. Next, in decision step 556, the AI ​​measurement control module 320 can determine whether a computer vision result has been received. For example, the AI ​​measurement control module 320 can determine whether the ROI detection module 306 has provided a computer vision result for the latest video frame in video frames 302a-302n. If the AI ​​measurement control module 320 has determined that a computer vision result has been received, method 550 can proceed to step 558. In step 558, the AI ​​measurement control module 320 can select an operating mode for generating AE parameters, which provides ROI tracking for AI measurement. Next, method 550 can proceed to step 566. In decision step 556, if the AI ​​measurement control module 320 has determined that a computer vision result has not yet been received, method 550 can proceed to decision step 560.

[0234] In decision step 560, the AI ​​metrology control module 320 can determine whether a computer vision result exists in the latest tracking loop number. In the example, the AI ​​metrology control module 320 can receive a historical timestamp signal HTS to determine whether any historical information in the tracking list queue 324 is a video frame from video frames 302a-302n within the tracking list number length provided by the signal TLN. If there is no historical information for a computer vision result within the tracking loop number, method 550 can move to step 562. In step 562, the AI ​​metrology control module 320 can select an operating mode for generating AE parameters, which provides measurement for the entire FOV. Next, method 550 can move to step 566. In decision step 560, if historical information for a computer vision result exists within the tracking loop number, method 550 can move to step 564. In step 564, the AI ​​metrology control module 320 can select an operating mode for generating AE parameters, which uses a metrology strategy with previous brightness values. Next, method 550 can move to step 566.

[0235] In step 566, the AI ​​metering control module 320 can generate the current luminance value CLUMA. Next, in step 568, the AE control module 322 can update the AE parameters. The signal AE_PARAM can be communicated to the capture device 104. Next, method 550 can proceed to step 570. Step 570 can end method 550.

[0236] refer to Figure 11 The diagram illustrates method (or process) 600. Method 600 can use full-field metrology to select automatic exposure parameters. Method 600 generally includes steps (or states) 602, 604, 606, 608, 610, decision steps (or states) 612, 614, 616, 618, decision steps (or states) 620, 622, and 624.

[0237] Step 602 can initiate method 600. In step 604, the AI ​​metering control module 320 can select an operating mode that provides the entire FOV metering. Next, in step 606, the tracking list queue 324 can clear any computer vision results in the tracking list queue 324 (e.g., all stored results may be outside the tracking loop number and may no longer be relevant to the incoming video frames 302a-302n). In step 608, the AI ​​metering control module 320 can receive the knee signal KNEE (e.g., user input), the metering weighting table signal MWT (user input and / or stored in memory 150), and / or the 3A statistics signal 3A_STAT (e.g., from IDSP 304). Next, in step 610, the AI ​​metering control module 320 can iterate to the next tile in tiles 402aa-402nn of one of the video frames 320a-320n. Next, method 600 can move to decision step 612.

[0238] In decision step 612, the AI ​​metering control module 320 can determine whether the 3A statistical data indicates that the current tile in tiles 402aa-402nn has a brightness value less than 2048. If the brightness value is less than 2048, method 600 can move to step 614. In step 614, the AI ​​metering control module 320 can scale the tile brightness value by 1 + knee. Next, method 600 can move to step 618. In decision step 612, if the brightness value is greater than or equal to 2048, method 600 can move to step 616. In step 616, the AI ​​metering control module 320 can scale the tile brightness value by 1 - knee. Next, method 600 can move to step 618.

[0239] In step 618, the AI ​​metering control module 320 can multiply the tile brightness by a tile weight (e.g., from a metering weighting table). Next, method 600 can move to decision step 620. In decision step 620, the AI ​​metering control module 320 can determine whether there are more tiles 402aa-402nn to be evaluated in the current video frame of video frames 302a-302n. If there are more tiles to be evaluated, method 600 can return to step 610. If there are no more tiles to be evaluated, method 600 can move to step 622. In step 622, the AI ​​metering control module 320 can calculate the average weighted brightness value. Next, method 600 can move to step 624. Step 624 can end method 600.

[0240] refer to Figure 12 The diagram illustrates method (or process) 650. Method 650 can select automatic exposure parameters using previous brightness values ​​without current computer vision results. Method 650 generally includes steps (or states) 652, 654, 656, decision step (or state) 658, 660, 662, 664, decision step (or state) 666, 668, 670, and 672.

[0241] Step 652 can initiate method 650. In step 654, the AI ​​metering control module 320 can select an operating mode that uses a metering strategy with previous brightness values. For example, no computer vision result is available for the current video frame in video frames 302a-302n. Next, in step 656, the AI ​​metering control module 320 can receive the next computer vision result from the tracking list queue 324 (e.g., the historical ROI information list signal HLROI). Method 650 can then proceed to decision step 658.

[0242] In decision step 658, the AI ​​measurement control module 320 can determine whether the computer vision result has been updated in the last tracking loop number of video frames 302a-302n. If there is no new computer vision result, method 650 can move to step 660. In step 660, the computer vision result can be removed from the tracking list queue 324. For example, the tracking list queue 324 can be updated if there is no new computer vision result. Next, method 650 can move to step 662. In decision step 658, if the computer vision result has been updated in the last tracking loop number of video frames 302a-302n, method 650 can move to step 662.

[0243] In step 662, the AI ​​metering control module 320 can use a brightness value from a previous computer vision result (e.g., presented as a signal CLUMA). For example, a brightness value from ROI information in the signal HLR0I, which has the most recent historical timestamp from the signal HTS, can be used. Next, in step 664, the AE control module 322 can reuse the previous brightness result as the current brightness value to compare with the target brightness value in order to generate AE parameters for the capture device 104. Next, method 650 can move to decision step 666.

[0244] In decision step 666, the AI ​​measurement control module 320 can determine whether a new computer vision result has been detected. In this example, the AI ​​measurement control module 320 can determine whether new ROI information has been provided in the signal LROI. If no new computer vision result has been received, method 650 can move to step 668. In step 668, the AI ​​measurement control module 320 can change to an operating mode that provides measurement of the entire FOV. Next, method 650 can move to step 672. In decision step 666, if a new computer vision result has been detected, method 650 can move to step 670. In step 670, the AI ​​measurement control module 320 can change to an operating mode that provides AI measurement with ROI tracking. Next, method 650 can move to step 672. Step 672 can end method 650.

[0245] refer to Figure 13The diagram illustrates method (or process) 700. Method 700 can use region of interest tracking to select automatic exposure parameters. Method 700 generally includes steps (or states) 702, 704, 706, 708, decision step (or state) 710, 712, 714, 716, 718, 720, 722, 724, 726, 728, and 730.

[0246] Step 702 can initiate method 700. In step 704, the AI ​​metering control module 320 can select an operating mode that provides AI metering with ROI tracking. Next, in step 706, the AI ​​metering control module 320 can receive ROI information for one of the ROIs 370a-370b and the next ROI ID in ROI IDs 380a-380b. In step 708, the AI ​​metering control module 320 can compare the difference between the timestamp TS of the current video frame in video frames 302a-302n from IDSP 304 and the updated timestamp TS' from ROI detection module 306. Next, method 700 can proceed to decision step 710.

[0247] In decision step 710, the AI ​​metering control module 320 can determine whether to perform ROI prediction. In the example, ROI prediction can be performed if the timestamp TS and the updated timestamp TS' do not match. If the AI ​​metering control module 320 is performing ROI prediction, method 700 can move to step 712. In step 712, the AI ​​metering control module 320 can compute a fitting function to estimate the predicted ROI location based on historical information stored in the tracking list queue 324. Next, method 700 can move to step 714. In decision step 710, if the AI ​​metering control module 320 is not performing ROI prediction, method 700 can move to step 714. In step 714, the AI ​​metering control module 320 can compare the coordinates of one of the ROI locations 384a-384b to classify each ROI tile in ROI tiles 412aa-412am. Next, method 700 can move to decision step 716.

[0248] In decision step 716, the AI ​​metering control module 320 can determine whether any valid ROI tiles exist. For example, if the ROI does not surround any of the tiles 402aa-402nn, then no valid ROI tiles may exist. If no valid ROI tiles exist, method 700 can move to step 718. In step 718, the AI ​​metering control module 320 can skip the average brightness calculation for the current ROI among ROIs 370a-370b. Next, method 700 can move to decision step 722. In decision step 716, if valid ROI tiles exist, method 700 can move to step 720. In step 720, the AI ​​metering control module 320 can apply 3A statistics and calculate the average brightness for the ROI. Next, method 700 can move to decision step 722.

[0249] In decision step 722, the AI ​​metering control module 320 can determine whether there are more ROIs in the ROI list provided by the ROI detection module 306. If there are more ROIs, method 700 can return to step 706 and repeat steps 706-722 for the ROI with the next ROI ID. If there are no more ROIs, method 700 can move to step 724. In step 724, the AI ​​metering control module 320 can apply ROI weights 382a-382b to calculate a weighted average brightness value for each ROI in ROIs 370a-370b and the background tiles. Next, in step 726, the AE control module 322 can update the AE parameters used for the capture device 102. In step 728, the AI ​​metering control module 320 can generate a historical timestamp signal HTS and a historical ROI information list signal HLROI to update the computer vision results in the tracking list queue 324. Next, method 700 can move to step 730. Step 730 can end method 700.

[0250] Depend on Figure 1-13The functions executed by the diagram can be implemented using one or more of a conventional general-purpose processor, digital computer, microprocessor, microcontroller, RISC (Reduced Instruction Set Computer) processor, CISC (Complex Instruction Set Computer) processor, SIMD (Single Instruction Multiple Data) processor, signal processor, central processing unit (CPU), arithmetic logic unit (ALU), video digital signal processor (VDSP), and / or similar computing machines programmed according to the teachings of this specification, as will be apparent to those skilled in the art. A skilled programmer can readily prepare appropriate software, firmware, code, routines, instructions, opcodes, microcode, and / or program modules based on the teachings of this disclosure, again as will be apparent to those skilled in the art. The software is typically executed from one or more media by one or more processors in a machine-implemented manner.

[0251] The present invention can also be implemented by preparing an ASIC (Application-Specific Integrated Circuit), a platform ASIC, an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), a CPLD (Complex Programmable Logic Device), a sea-of-gate, an RFIC (Radio Frequency Integrated Circuit), an ASSP (Application-Specific Standard Product), one or more monolithic integrated circuits, one or more chips or dies arranged as flip-chip modules and / or multi-chip modules, or by interconnecting conventional component circuits through a suitable network, as described herein, modifications of which will be apparent to those skilled in the art.

[0252] Therefore, the present invention may also include a computer product, which may be one or more storage media and / or one or more transmission media, including instructions that can be used to program a machine to perform one or more processes or methods according to the present invention. The execution of the instructions contained in the computer product by the machine, and the operation of surrounding circuitry, can convert input data into one or more files on the storage medium and / or one or more output signals representing physical objects or substances, such as audio and / or visual depictions. The storage medium may include, but is not limited to, any type of disk, including floppy disks, hard disks, magnetic disks, optical disks, CD-ROMs, DVDs, and magneto-optical disks, as well as circuits such as ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), UVPROM (Ultraviolet Erasable Programmable ROM), flash memory, magnetic cards, optical cards, and / or any type of medium suitable for storing electronic instructions.

[0253] The elements of this invention can form part or all of one or more devices, units, components, systems, machines, and / or apparatuses. Devices may include, but are not limited to, servers, workstations, storage array controllers, storage systems, personal computers, laptop computers, notebook computers, handheld computers, cloud servers, personal digital assistants, portable electronic devices, battery-powered devices, set-top boxes, encoders, decoders, transcoders, compressors, decompressors, preprocessors, post-processors, transmitters, receivers, transceivers, cryptographic circuits, cellular phones, digital cameras, positioning and / or navigation systems, medical devices, head-up displays, wireless devices, audio recording, audio storage and / or audio playback devices, video recording, video storage and / or video playback devices, gaming platforms, peripheral devices, and / or multi-chip modules. Those skilled in the art will understand that the elements of this invention can be implemented in other types of devices to meet specific application standards.

[0254] When used herein in conjunction with "is" and verbs, the terms "may" and "usually" are intended to convey the intention that the description is exemplary and is considered broad enough to encompass both the specific examples presented in this disclosure and the alternative examples that may be derived based on this disclosure. As used herein, the terms "may" and "usually" should not be construed as necessarily implying the desirability or possibility of omitting the corresponding element.

[0255] When the notations “a”-“n” are used herein for various components, modules, and / or circuits, a single component, module, and / or circuit or multiple such components, modules, and / or circuits are disclosed, wherein the notation “n” is applied to represent any particular integer. Each different component, module, and / or circuit having an instance (or event) notated as “a”-“n” may indicate that the different component, module, and / or circuit may have a matching number of instances or a different number of instances. An instance designated as “a” may represent the first of multiple instances, and an instance “n” may refer to the last of multiple instances without implying a specific number of instances.

[0256] Although the invention has been specifically shown and described with reference to embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention.

Claims

1. An apparatus comprising: An interface configured to receive pixel data; as well as A processor configured to: (i) process the pixel data arranged as video frames; (ii) Generate statistics and timestamps for the video frames; (iii) Perform a computer vision operation on the video frames to determine one or more regions of interest (ROIs) within the video frames at the timestamps, wherein the computer vision operation is configured to generate information about a list of the ROIs and a modified timestamp, and the modified timestamp includes the timestamp plus an amount of time used to perform the computer vision operation; (iv) Track the movement of the ROIs over a sequence of video frames; (v) Perform a prediction of the location of the ROIs in response to determining that the timestamps and the modified timestamps do not match, and evaluate the current brightness value of the ROIs based on the prediction; (vi) Calculate automatic exposure parameters in response to the current brightness value and a target brightness value; and (vii) Generate an output signal including the automatic exposure parameters, wherein the automatic exposure parameters are calculated for: (i) Prioritize the exposure of the region of interest in the video frame, and (ii) Dynamically change the automatic exposure parameters based on the movement of the region of interest.

2. The apparatus according to claim 1, wherein, The automatic exposure parameters are dynamically changed so that, in response to changes in the current brightness value due to various lighting conditions, the exposure in the region of interest within the video frame is made consistent across the sequence of video frames.

3. The apparatus according to claim 1, wherein, The statistics include information from the video frames regarding automatic white balance, automatic focus, and automatic exposure.

4. The apparatus according to claim 1, wherein, Each entry in the list of regions of interest includes a position, an identifier value, and a weighted value.

5. The apparatus according to claim 4, wherein, The computer vision operation is configured to track the movement of the region of interest over the sequence of video frames based on the identifier value.

6. The apparatus according to claim 4, wherein, The weighting values ​​(i) include values ​​between 0.0 and 1.0, and (ii) provide priority for the region of interest in the automatic exposure parameters.

7. The apparatus according to claim 4, wherein, The current brightness value of the region of interest is evaluated in response to the information about the region of interest, the modified timestamp, the timestamp, and the statistics.

8. The apparatus according to claim 7, wherein, The current brightness value of the region of interest is further evaluated in response to (i) tracking cycle number, (ii) metering weighting table and (iii) initial setting of weighting values, in order to balance the exposure between bright and dark areas of the video frame.

9. The apparatus according to claim 1, wherein, The prediction of the location of the region of interest is determined in response to the current location of the region of interest at the modified timestamp and historical information about the region of interest stored in the tracking list queue.

10. The apparatus according to claim 1, wherein, The computer vision operations include one or more of the following: person detection, vehicle detection, motion detection, and face detection.

11. The apparatus according to claim 1, wherein, Each region of interest corresponds to a detected object as defined by the computer vision operation.

12. The apparatus according to claim 1, wherein, The output signal is provided to the image sensor of the capture device.

13. The apparatus according to claim 12, wherein, The automatic exposure parameters include one or more of the following: the shutter speed, aperture value, and automatic gain control of the image sensor.

14. The apparatus according to claim 1, wherein, The processor is configured to calculate the automatic exposure parameters for the entire field of view of the video frame in response to (i) the computer vision operation not detecting at least one region of interest in the region of interest, and (ii) the absence of a prior history of the region of interest.

15. The apparatus according to claim 1, wherein, The processor is configured to reuse the current luminance value from the previous video frame in response to (i) the computer vision operation not detecting at least one region of interest in the region of interest in the current video frame of the video frame, and (ii) having a previous history of the region of interest in the previous video frame of the video frame.

16. The apparatus according to claim 15, wherein, If the current brightness value corresponds to the previous history of the region of interest within the tracking loop number of the video frame, then the current brightness value from the previous video frame in the video frame is used.

17. The apparatus according to claim 15, wherein, The previous history of a predetermined number of the regions of interest is stored in a tracking list queue.

18. The apparatus according to claim 1, wherein, The statistics include a table of tile brightness values ​​corresponding to multiple tiles of the video frame.

Images