Dynamic actuator mechanism for rolling shutter sensor to obtain structured light patterns

Abstract

A device includes a rolling shutter image sensor, a structured light projector, and a processor. The rolling shutter image sensor may be configured to capture an image. The structured light projector may be configured to generate a structured light pattern. The processor may be configured to: receive an image from the rolling shutter image sensor; perform computer vision operations on the image to detect an object in the image; determine a start line position and an end line position corresponding to the detected object; and present a control signal to the structured light projector. The control signal may be configured to: activate the structured light pattern when the rolling shutter image sensor is at the start line position; and deactivate the structured light pattern when the rolling shutter image sensor is at the end line position.

Description

Technical Field

[0001] The present invention relates generally to 3D sensing systems, and more particularly to a method and / or apparatus for implementing a dynamic actuator mechanism for a rolling shutter sensor to acquire structured light patterns. Background Art

[0002] Conventional 3D sensing solutions utilize global shutter sensors to capture structured light patterns. Global shutter sensors are high-cost devices because all pixels of a global shutter sensor begin to be exposed at the same time. Using global shutter sensors prevents low-cost devices from being a solution in 3D sensing applications.

[0003] Rolling shutter sensors are not typically used to capture structured light patterns because they cannot expose all pixels of the sensor at the same time. Rolling shutter sensors are often lower cost sensors compared to global shutter sensors. Rolling shutter sensors expose only a portion of the sensor's pixels at a time (i.e., each row (or line) of pixels begins exposure at a different time, typically from the top of the sensor to the bottom of the sensor over time). Acquiring a structured light pattern with a rolling shutter sensor at a given frame will affect other frames acquired at different times. If the structured light pattern cannot be matched with an infrared (IR) frame at the same time, the IR frame may have only half or less of the frames with the structured light pattern. Even if a structured light pattern is present, the pattern will not be clear enough because the projector is only turned on for a short time due to hardware limitations.

[0004] It is desirable to implement a dynamic actuator mechanism for a rolling shutter sensor to acquire structured light patterns. Summary of the invention

[0005] The present invention relates to an apparatus including a rolling shutter image sensor, a structured light projector, and a processor. The rolling shutter image sensor may be configured to capture an image. The structured light projector may be configured to generate a structured light pattern. The processor may be configured to: receive an image from the rolling shutter image sensor; perform computer vision operations on the image to detect an object in the image; determine a start line position and an end line position corresponding to the detected object; and present a control signal to the structured light projector. The control signal may be configured to: activate the structured light pattern when the rolling shutter image sensor is at the start line position; and deactivate the structured light pattern when the rolling shutter image sensor is at the end line position. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Embodiments of the present invention will become apparent from the following detailed description, appended claims and accompanying drawings.

[0007] Figure 1is a diagram illustrating a rolling shutter sensor system according to an example embodiment of the present invention.

[0008] Figure 2 is a diagram showing a block diagram of a camera system.

[0009] Figure 3 is a diagram illustrating a rolling shutter system in a camera system including an RGB-IR rolling shutter sensor.

[0010] Figure 4 is a diagram illustrating a rolling shutter system in a camera system including a separate IR rolling shutter sensor and an RGB rolling shutter sensor.

[0011] Figure 5 is a diagram illustrating elements of a 3D sensing system according to an example embodiment of the present invention.

[0012] Figure 6 is a diagram showing the start and end lines of detecting a region of interest in an image frame.

[0013] Figure 7 is a diagram showing the start time and end time of exposure for an IR image frame.

[0014] Figure 8 is a diagram showing a timing chart for sensor exposure.

[0015] Fig. 9 is a diagram illustrating an example security panel providing a facial disarm feature.

[0016] Fig.10 is a diagram illustrating an example access control panel.

[0017] Fig.11 is a diagram showing an example access to a control panel using a password.

[0018] Fig.12 is a diagram illustrating an example access control panel using facial recognition.

[0019] Fig.13 is a flow chart illustrating a method for acquiring a structured light pattern using a rolling shutter sensor.

[0020] Fig.14 is a flow chart illustrating a method for performing computer vision to detect objects to determine start and end lines of a region of interest in a video frame when a structured light pattern is inactive.

[0021] Fig.15 is a flow chart illustrating a method for performing depth analysis when a structured light pattern is active.

[0022] Fig.16is a flow chart illustrating a method for implementing a dynamic actuator mechanism for a rolling shutter sensor. DETAILED DESCRIPTION

[0023] Embodiments of the present invention include providing a dynamic driver mechanism for a rolling shutter sensor to acquire a structured light pattern, which dynamic driver mechanism can (i) control the shutter exposure time of the rolling shutter sensor; (ii) control the on time and duration of a structured light projector; (iii) detect an object of interest to determine a region of interest; (iv) utilize a signal (or interrupt) from the rolling shutter sensor as a trigger signal; (v) generate a separate signal in response to the trigger signal to control the structured light projector; (vi) utilize a timer to control the on time of the structured light projector; (vii) align the timing of the structured light with the timing of the exposed pixels of the rolling shutter sensor corresponding to the region of interest; (viii) have a lower cost compared to a global shutter sensor implementation; (ix) provide greater flexibility for controlling the structured light projector; (x) have reduced power consumption compared to a global shutter sensor implementation; (xi) be implemented in a low-power and battery-operated device; (xii) be used to implement a low-cost structured light-based 3D sensing system; and / or (xiii) be implemented as one or more integrated circuits.

[0024] Embodiments of the present invention may implement a dynamic driver mechanism for a rolling shutter sensor. The dynamic driver mechanism may be configured to acquire a structured light pattern (SLP). A rolling shutter sensor may not be able to expose all pixels of the sensor at the same time. The dynamic driver mechanism may be configured to enable the rolling shutter sensor to acquire a structured light pattern at a specific area. By enabling the rolling shutter sensor to acquire a structured light pattern at a specific area, a depth image may be generated using the structured light pattern. For example, the depth image may be used for 3D sensing, 3D modeling, 3D facial recognition, presence judgment, and the like.

[0025] In various embodiments, a low-cost 3D sensing platform can be implemented that includes a vision system-on-chip (SoC), a structured light projector, and an RGB-IR rolling shutter image sensor. In various embodiments, a single RGB-IR image sensor can be used to obtain both visible light images and infrared (IR) images. In various embodiments, visible light images (without structured light patterns) and / or infrared images (without structured light patterns) can be used for viewing, object (e.g., face, etc.) detection, object identification, and / or facial recognition. Infrared (IR) images with structured light patterns can be used for depth sensing and presence determination. For example, an IR image without a structured light pattern can be generated (e.g., when the SLP projector is in an off state), and the IR image without a structured light pattern can also be used for facial detection or facial recognition. However, an IR image with a structured light pattern can be used for depth sensing / 3D sensing / presence determination, while an IR image without a structured light pattern may not be able to perform depth sensing / 3D sensing / presence determination. In an example, a visual SoC may provide depth processing, anti-spoofing, structured light control, object detection, object identification and / or classification, 3D facial detection and / or recognition, and video encoding on a single chip. In an example, a visual SoC may generate one or more control signals in response to analyzing an image captured by an RGB-IR rolling shutter image sensor. In an example, a visual SoC may use one or more control signals to control one or more features of a security system (e.g., door locks, alarm systems, controlled access systems, payment systems, etc.). In an example, a visual SoC may be configured to store and execute an artificial neural network (ANN), thereby enabling feature detection and extraction, object detection and identification, and facial detection and recognition. In an example, an ANN may be stored (e.g., in a non-volatile computer-readable storage medium) as a directed acyclic graph (DAG) and corresponding weights.

[0026] Embodiments of the present invention may be configured to crop a portion of an IR image and focus on a specified region of interest (ROI) to acquire a structured light pattern. The acquisition of the structured light pattern may be dynamic to ensure that the ROI can be filled with the structured light pattern on the infrared (IR) image (e.g., the structured light pattern has sufficient integration time for the ROI). In an example, the ROI may be a location, region, and / or portion of an image that includes an object of interest (e.g., a face). A depth image may be calculated based on the IR image with the structured light pattern.

[0027] Embodiments of the present invention may be configured to turn on the structured light from the start line of the ROI until the end line of the ROI. The ROI may be dynamically changed based on the positioning of the object of interest. If the activation timing of the structured light at a specified line of the ROI does not match the IR frame exposure timing at the same time, the ROI may not have a structured light with sufficient integration time. A fixed interrupt signal and a system clock-based timer may be implemented to control the timing to prevent a mismatch between the IR frame exposure timing and the ROI acquisition timing of the rolling shutter sensor. Embodiments of the present invention may be configured to use a common rolling shutter sensor (e.g., separate RGB and IR sensors) and / or an RGB-IR rolling shutter sensor (e.g., using a single sensor to receive both RGB and IR pixel data).

[0028] Embodiments of the present invention may be configured to implement a convolutional neural network (CNN). Computer vision operations may be performed to enable the CNN to detect the presence of an object of interest. The location (e.g., coordinates) of the object of interest may be detected in an image frame. For example, a face detection operation may be used on an image frame to detect a face. The location of the object of interest may be used to determine the start and end lines of a region of interest in an image frame. Structured light may be dynamically activated based on the start and end lines of the region of interest (e.g., a structured light pattern in the ROI may be acquired using a facial location determined in an image frame).

[0029] Since rolling shutters may be less expensive than global shutter image sensors, implementing embodiments of the present invention may enable 3D sensing in low-cost devices. The timing of the structured light projector may be controllable. The timing sequence may be easily adjusted for flexible design (e.g., the timing may be corrected based on feedback). Controlling the timing of activation of the structured light may provide power savings. The structured light pattern may be turned off when an object of interest is not present.

[0030] Embodiments of the present invention may be used in low-cost 3D sensing devices. 3D sensing may be used in a variety of applications to detect and / or identify a person's face. In one example, a device may use facial recognition to unlock a door and / or disarm an alarm system. In another example, a device may use facial recognition to allow access to a specific area (e.g., "tripwire" access to a restricted area). For example, it may be recognized that a gardener / pool maintenance person is entering the backyard of a residential room, and the room security alarm may not be triggered (e.g., this may be limited to certain times and days of the week). In yet another example, facial recognition may be used to trigger an alarm upon recognition (e.g., if a restraining order is against an ex-spouse, an alarm may be issued to emergency services if the ex-spouse is detected). In yet another example, a security system may be accessed by obtaining privileges based on video / audio identification (e.g., similar to superuser access in Linux, where only specific people or groups are allowed to change access levels or policies, add users, etc., even if the correct password is entered). The type of device implementing the present invention may vary according to the design criteria of a particular implementation.

[0031] A low-cost 3D sensing platform according to an embodiment of the present invention can be configured to implement accurate biometric identification that also maintains user privacy. Accurate biometric identification can enable smart access control and video security products with facial biometrics and presence detection for racially diverse populations. Biometric identification can enable contactless access (e.g., suitable for use during epidemics). Sensing using 3D rather than 2D recognition can prevent spoofing, false positives, false negatives, gender bias, racial bias, etc. For example, depth processing, anti-spoofing, 3D facial recognition, and video encoding can be implemented on a single processing chip.

[0032] In various applications, a low-cost 3D sensing platform according to an embodiment of the present invention can significantly reduce system complexity while improving performance, reliability, and safety. In an example, a visual SoC according to an embodiment of the present invention may include, but is not limited to, a powerful image signal processor (ISP), native support for RGB-IR color filter arrays, and advanced high dynamic range (HDR) processing, which can produce excellent image quality in low-light and high-contrast environments. In an example, a visual SoC according to an embodiment of the present invention can provide an architecture that delivers computing power for presence detection and 3D facial recognition while running multiple artificial intelligence (AI) models for advanced features such as people counting and anti-tailgating. Various embodiments of the present invention can implement a low-cost 3D sensing platform, as described in co-pending U.S. patent application No. 16 / 996,114 filed on August 18, 2020, appropriate portions of which are incorporated herein by reference.

[0033] refer to Figure 1 , a block diagram of an apparatus illustrating an example implementation of a dynamic driver mechanism according to an example embodiment of the present invention is shown. A system 100 is shown. The system 100 can implement a camera system and / or a 3D sensing platform. In one example, the system 100 can include a dynamic driver mechanism for an RGB-IR rolling shutter sensor that is configured to acquire a structured light pattern. In another example, the system 100 can include a dynamic driver mechanism for an IR rolling shutter sensor (e.g., an IR sensor implemented separately from an RGB image sensor).

[0034] System 100 may include block (or circuit) 102, block (or circuit) 104, and / or block (or circuit) 106. Circuit 102 may implement a processor and / or a system on a chip (SoC). Circuit 104 may implement an infrared structured light projector. Circuit 106 may implement a capture device (or module). In an example, capture device 106 may be a security / surveillance camera. System 100 may include other components (not shown). The number, type, and / or arrangement of the components of system 100 may vary according to the design criteria of a particular implementation.

[0035] Processor 102 may include block (or circuit) 108 and / or block (or circuit) 110. Circuit 108 may implement an image signal processing (ISP) circuit (or processor or front end). In one example, circuit 108 is generally capable of performing multi-channel ISP. Circuit 110 may implement a control circuit (e.g., a dedicated circuit, an embedded controller, a processor, a microprocessor, etc.). In an example, circuit 102 may include block (or circuit) 112. Circuit 112 may implement a structured light (SL) control circuit (or function). In one example, circuit 110 and circuit 112 may be implemented as separate circuit cores, which may be instantiated on a single integrated circuit substrate (or die) or in a multi-chip module (MCM). Processor 102 may include other components (not shown). For example, processor / SoC 102 may include memory, an audio processor, and / or other hardware modules. The number, type, and / or arrangement of components of processor 102 may vary according to the design criteria of a particular implementation.

[0036] The structured light projector 104 may include a structured light source 120. The structured light source 120 may be configured to generate a signal (eg, SLP). The signal SLP may be a structured light pattern.

[0037] The capture device 106 may include a rolling shutter sensor 130. In the example shown, the rolling shutter sensor 130 may implement an RGB-IR sensor. In some embodiments, the capture device 106 may include a rolling shutter IR sensor and an RGB sensor (e.g., implemented as separate components). The capture device 106 may be configured to receive an input signal (e.g., LIN). The signal LIN may be an optical input. The rolling shutter sensor 130 may be configured to convert the input light LIN into computer readable data.

[0038] In an example, the structured light source 120 can be implemented as an array of vertical cavity surface emitting lasers (VCSELs) and a lens. However, other types of structured light sources can be implemented to meet the design criteria of a specific application. In an example, the array of VCSELs is generally configured to generate a laser pattern (e.g., signal SLP). The lens is generally configured to decompose the laser pattern into a dense array of dot patterns. In an example, the structured light source 120 can implement a near-infrared (NIR) light source. In various embodiments, the light source of the structured light source 120 can be configured to emit light with a wavelength of approximately 940 nanometers (nm), which is invisible to the human eye. However, other wavelengths can be utilized. In an example, a wavelength in the range of approximately 800nm-1000nm can be utilized.

[0039] In various embodiments, the circuit 110 may be connected to the IR structured light projector 104, the capture device 106, and the ISP circuit 108. The capture device 106 may also be connected to the ISP circuit 108. In an example, the circuit 110 generally provides a central control mechanism to synchronize the timing of the structured light projector 104 and the capture device 106. In an example, the circuit 110 may be configured to calculate and maintain a predefined timing model to control the structured light source 120 of the structured light projector 104. In an example, the circuit 110 may also be configured to control the exposure time of the rolling shutter sensor 130 of the capture device 106. In an example, the circuit 110 may also be configured to control the ISP circuit 108 to synchronize with the output of the capture device 106.

[0040] In various embodiments, circuit 110 may be configured to generate one or more video output signals (e.g., VIDOUT) and one or more control signals (e.g., CTRL). In an example, the one or more control signals CTRL may be used to control features (or operations) of one or more devices external to circuit 110 and / or SoC 102.

[0041] In some embodiments, the image signal processor 108 may be configured to present a signal (e.g., EOF). The signal EOF may be an interrupt signal. The interrupt signal EOF may provide an indication of when the first row (or first line) of the rolling shutter sensor 130 begins exposure (or provide information that facilitates calculation using a predefined formula). The interrupt signal EOF may represent an indication of the end of a frame generated by the rolling shutter sensor 130. The indication of the end of a frame may correspond to the end of a previous frame. The end time of the previous frame may be used to calculate when the first line begins exposure of the current frame. The interrupt signal EOF may be a soft interrupt (e.g., not connected to a hardware pin). In one example, the image signal processor 108 (or IDSP) may be configured to generate the interrupt signal EOF. In another example, other sensor signals from the capture device 106 may be used to calculate (e.g., using a predefined formula, etc.) when the first line begins exposure.

[0042] The circuit 112 may utilize the interrupt signal EOF from the image signal processor 108 to control the structured light projector 104. In another example, the interrupt signal EOF may be configured to generate an indication that the first row of the rolling shutter sensor 130 may begin exposure. The interrupt may cause the circuit 112 to start a predefined turn-on period of the structured light source 120 of the structured light projector 104. In an example, the circuit 112 may be configured to program a timer with a predefined turn-on period. In response to receiving the interrupt signal EOF, the circuit 112 may start a timer to turn on the structured light source for a predefined period of time.

[0043] In the example shown, the circuit 110 may have an input that may receive the interrupt signal EOF. In the example shown, the circuit 110 may have an input / output that may communicate with an input / output of the ISP circuit 108 via a signal (e.g., ISP SYNC). The signal ISP SYNC may be configured to synchronize the timing of the structured light projector 104 with the capture device 106.

[0044] In the example shown, circuit 110 may have an input / output that can communicate a first image channel (e.g., RGB DATA) with an input / output of ISP circuit 108 and another input / output that can communicate a second image channel (e.g., IR DATA) with another input / output of ISP circuit 108.

[0045] In the example shown, circuit 110 may have an output that may present a signal (e.g., SL TRIG) to an input of structured light projector 104. Circuit 110 may also include an output for presenting one or more video output signals VIDOUT and / or another output that may present one or more control signals CTRL. Circuit 104 may be configured to generate a structured light pattern in response to signal SL TRIG.

[0046] In the example shown, the capture device 106 can receive a light input LIN. In the example shown, the capture device 106 can have an output that can present data that can communicate information that can be used to calculate the start time of the first line of exposure. For example, when the start time of the first line of exposure is known, the processor 102 can decide when to turn on the structured light source 120 at a specified time to ensure that the top of the ROI has a structured light pattern from the bottom of the ROI.

[0047] The capture device 106 may also include an input / output (e.g., VIDEO) that may communicate signals to / from the input / output of the ISP circuit 108. In an example, the signal VIDEO may communicate four channels of video pixel information (e.g., R, G, B, and IR) to the ISP circuit 108. In one example, the capture device 106 and the ISP circuit 108 may also exchange control and / or status signals (e.g., bidirectional communication) via a connection carrying the signal VIDEO. The capture device 106 may present the signal VIDEO to the image signal processor 108. The signal VIDEO may include image data from RGB and / or IR pixels. When the image signal processor 108 receives the signal VIDEO, the image signal processor 108 may analyze the frame timing and / or determine the end of a frame and generate an interrupt signal EOF.

[0048] In an example, the ISP circuit 108 may be configured to separate a four-channel RGB-IR video signal VIDEO into separate IR image data channels and RGB image data channels. In an example, the ISP circuit 108 may be configured to generate a first image channel RGB DATA and a second image channel IR DATA in response to the signal VIDEO. The first image channel RGB DATA typically includes color image data. In an example, the color image data may include RGB color space data or YUV color space data. In an example, the second image channel IR DATA may include monochrome image data. When a structured light pattern is projected by the structured light projector circuit 104, the second image channel IR DATA typically includes both the IR image data and the structured light pattern projected by the structured light projector circuit 104. When the structured light pattern is not projected by the structured light projector circuit 104, the second image channel IR DATA typically includes IR image data without the structured light pattern. When the structured light pattern is projected by the structured light projector circuit 104, the first image channel RGB DATA typically also includes the structured light pattern projected by the structured light projector circuit 104, and is therefore typically ignored.

[0049] In an example, the structured light pattern data carried by the second image channel IR DATA may be analyzed by the circuit 102 to obtain 3D (e.g., depth) information for the field of view of the capture device 106. The circuit 110 may also be configured to make a presence determination based on the structured light pattern data carried by the second image channel IR DATA. In an example, the circuit 110 may analyze the RGB (or YUV) data and the IR data without the structured light pattern to discern (e.g., detect, identify, etc.) one or more features or objects in the field of view of the capture device 106.

[0050] In an example, the circuit 112 may be configured to generate the signal SL TRIG. The circuit 112 may implement the structured light control timing protocol according to an embodiment of the present invention (hereinafter combined with Figure 2-8 In an example, the circuit 112 may be implemented in hardware, software (or firmware, microcode, etc.), or a combination of hardware and software.

[0051] In an example, the structured light source 120 may be configured to emit a structured light pattern in response to the signal SL TRIG. In an example, the period and / or intensity of light emitted by the structured light source 120 may be controlled (e.g., programmed) by the circuit 110. In an example, the circuit 110 may configure the structured light source 120 before activating the signal SL TRIG.

[0052] In an example, the rolling shutter sensor 130 may be implemented as a rolling shutter image sensor. In an example, the rolling shutter sensor 130 may be implemented as an RGB-IR rolling shutter complementary metal oxide semiconductor (CMOS) image sensor. In one example, the rolling shutter sensor 130 may be configured to enable a signal (e.g., FLES, not shown) indicating a first row exposure time. In another example, the rolling shutter sensor 130 may be configured to provide a signal VIDEO, and the image signal processor 108 may calculate the start of exposure of the first row (or row) of the rolling shutter sensor 130 using a predefined formula (in response to detecting the end of the previous frame, so as to indicate that the first row (or row) of the rolling shutter sensor 130 starts exposure for the current image frame).

[0053] In one example, the rolling shutter sensor 130 can apply a mask to a monochrome sensor. In an example, the mask can include multiple cells that include a red pixel, a green pixel, a blue pixel, and an IR pixel. The IR pixel can include red, green, and blue filter materials that can efficiently absorb all light in the visible spectrum while allowing longer infrared wavelengths to pass with minimal loss. With a rolling shutter, as each row (or row) of the sensor begins to be exposed, all pixels in that row (or row) can begin to be exposed at the same time.

[0054] refer to Figure 2, a block diagram of a camera system 100 illustrating an example implementation is shown. In an example, the camera system 100 may include a processor / SoC 102, a structured light projector 104, a capture device 106, a block (or circuit) 150, a block (or circuit) 152, a block (or circuit) 154, a block (or circuit) 156, a block (or circuit) 160, a block (or circuit) 162, a block (or circuit) 164, and / or a block (or circuit) 166. Circuit 150 may implement a memory. Circuit 152 may implement a battery 152. Circuit 154 may implement a communication device. Circuit 156 may implement a wireless interface. Block 160 may implement an optical lens. Block 162 may implement a structured light pattern lens. Circuit 164 may implement one or more sensors. Circuit 166 may implement a human interface device (HID). In some embodiments, the camera system 100 may include a processor / SoC 102, an IR structured light projector 104, a capture device 106, a memory 150, a lens 160, an IR structured light projector 104, a structured light pattern lens 162, a sensor 164, a battery 152, a communication module 154, and a wireless interface 156. In another example, the camera system 100 may include a processor / SoC 102, a structured light projector 104, a capture device 106, a lens 160, a structured light pattern lens 162, and a sensor 164, and the memory 150, the battery 152, the communication module 154, and the wireless interface 156 may be components of a separate device. The camera system 100 may include other components (not shown). The number, type, and / or arrangement of the components of the camera system 100 may vary according to the design criteria of a particular implementation.

[0055] The memory 150 may store data. The memory 150 may implement various types of memory, including but not limited to cache, flash memory, memory card, random access memory (RAM), dynamic RAM (DRAM) memory, etc. The type and / or size of the memory 150 may vary according to the design criteria of a particular implementation. The data stored in the memory 150 may correspond to video files, motion information (e.g., readings from the sensor 164), video fusion parameters, image stabilization parameters, user input, computer vision models, and / or metadata information.

[0056] Processor / SoC 102 may be configured to execute computer readable code and / or process information. In various embodiments, computer readable code may be stored in processor / SoC 102 (e.g., microcode, etc.) and / or in memory 150. In an example, processor / SoC 102 may be configured to execute one or more artificial neural network models (e.g., facial recognition CNN, object detection CNN, object classification CNN, etc.) stored in memory 150. In an example, memory 150 may store one or more directed acyclic graphs (DAGs) and one or more weight sets defining one or more artificial neural network models. Processor / SoC 102 may be configured to receive input from memory 150 and / or present output to memory 150. Processor / SoC 102 may be configured to present and / or receive other signals (not shown). The number and / or type of input and / or output of processor / SoC 102 may vary according to the design criteria of a particular implementation. Processor / SoC 102 may be configured for low power (e.g., battery) operation.

[0057] Battery 152 may be configured to store and / or supply power to components of camera system 100. A dynamic actuator mechanism for rolling shutter sensor 130 may be configured to conserve power consumption. Reducing power consumption may enable camera system 100 to operate for an extended period of time using battery 152 without requiring recharging.

[0058] The communication module 154 may be configured to implement one or more communication protocols. For example, the communication module 154 and the wireless interface 156 may be configured to implement one or more of the following: IEEE 102.11, IEEE 102.15, IEEE 102.15.1, IEEE 102.15.2, IEEE 102.15.3, IEEE 102.15.4, IEEE 102.15.5, IEEE 102.20, and / or In some embodiments, wireless interface 156 may also implement one or more protocols associated with a cellular communication network (e.g., GSM, CDMA, GPRS, UMTS, CDMA2000, 3GPP LTE, 4G / HSPA / WiMAX, SMS, etc.). In embodiments where camera system 100 is implemented as a wireless camera, the protocols implemented by communication module 154 and wireless interface 156 may be wireless communication protocols. The type of communication protocol implemented by communication module 154 may vary depending on the design criteria of a particular implementation.

[0059] The communication module 154 and / or the wireless interface 156 may be configured to generate a broadcast signal as an output from the camera system 100. The broadcast signal may send the video data VIDOUT and / or the control signal(s) CTRL to an external device. For example, the broadcast signal may be sent to a cloud storage service (e.g., a storage service that can be expanded on demand). In some embodiments, the communication module 154 may not send data until the processor / SoC 102 has performed video analysis to determine that an object is in the field of view of the camera system 100.

[0060] In some embodiments, communication module 154 may be configured to generate a manual control signal. The manual control signal may be generated in response to a signal from a user received by communication module 154. The manual control signal may be configured to activate processor / SoC 102. Processor / SoC 102 may be activated in response to the manual control signal regardless of the power state of camera system 100.

[0061] The lens 160 may be attached to the capture device 106. In an example, the capture device 106 may include an image sensor 130, a block (or circuit) 182, and a block (or circuit) 184. The circuit 182 may be a processor and / or logic. The circuit 184 may be a memory circuit (e.g., a frame buffer). The lens 160 (e.g., a camera lens) may be oriented to provide a view of the environment surrounding the camera system 100. The lens 160 may be intended to capture environmental data (e.g., light input LIN). The lens 160 may be a wide-angle lens and / or a fisheye lens (e.g., a lens capable of capturing a wide field of view). The lens 160 may be configured to capture and / or focus light for the capture device 106. Typically, the image sensor 130 is located behind the lens 160. Based on the light captured from the lens 160, the capture device 106 may generate a bitstream and / or video data (e.g., signal VIDEO).

[0062] The capture device 106 can be configured to capture video image data (e.g., light collected and focused by the lens 160). The capture device 106 can capture data received through the lens 160 to generate a video bitstream (e.g., a sequence of video frames). In various embodiments, the lens 160 can be implemented as a fixed focus lens. Fixed focus lenses generally promote smaller size and low power. In an example, a fixed focus lens can be used for battery-powered doorbells and other low-power camera applications. In some embodiments, the lens 160 can be oriented, tilted, panned, zoomed, and / or rotated to capture the environment around the camera system 100 (e.g., capture data from a field of view). In an example, an active lens system can be utilized to implement a professional camera model for enhanced functionality, remote control, and the like.

[0063] The capture device 106 may convert the received light into a digital data stream. In some embodiments, the capture device 106 may perform analog-to-digital conversion. For example, the image sensor 130 may perform a photoelectric conversion on the light received by the lens 160. The processor / logic 182 may convert the digital data stream into a video data stream (or bit stream), a video file, and / or a plurality of video frames. In an example, the capture device 106 may present the video data as a digital video signal (e.g., VIDEO). The digital video signal may include video frames (e.g., sequential digital images and / or audio).

[0064] The video data captured by the capture device 106 may be represented as a signal / bitstream / data VIDEO (e.g., a digital video signal). The capture device 106 may present the signal VIDEO to the processor / SoC 102. The signal VIDEO may represent a video frame / video data. The signal VIDEO may be a video stream captured by the capture device 106. In some embodiments, the signal VIDEO may include pixel data that may be operated on by the processor 102 (e.g., a video processing pipeline, ISP 108, etc.). The processor 102 may generate a video frame in response to the pixel data in the signal VIDEO.

[0065] The image sensor 130 may receive input light LIN from the lens 160 and convert the input light LIN into digital data (e.g., a bit stream). For example, the image sensor 130 may perform a photoelectric conversion on the light from the lens 160. In some embodiments, the image sensor 130 may have an additional margin that is not used as part of the image output. In some embodiments, the image sensor 130 may not have an additional margin. In various embodiments, the image sensor 130 may be configured to generate an RGB-IR video signal. In a field of view illuminated only by infrared light, the image sensor 130 may generate a monochrome (B / W) video signal. In a field of view illuminated by both IR light and visible light, the image sensor 130 may be configured to generate color information in addition to the monochrome video signal. In various embodiments, the image sensor 130 may be configured to generate a video signal in response to visible light and / or infrared (IR) light.

[0066] The processor / logic 182 may convert the bitstream into human-viewable content (e.g., video data that can be understood by an average person regardless of image quality, such as video frames and / or pixel data that can be converted by the processor 102 into video frames). For example, the processor / logic 182 may receive pure (e.g., raw) data from the image sensor 130 and generate (e.g., encode) video data (e.g., a bitstream) based on the raw data. The capture device 106 may have a memory 184 to store the raw data and / or the processed bitstream. For example, the capture device 106 may implement a frame memory and / or a buffer 184 to store one or more of the video frames (e.g., digital video signals) (e.g., to provide temporary storage and / or caching). In some embodiments, the processor / logic 182 may perform analysis and / or correction on the video frames stored in the memory / buffer 184 of the capture device 106. The processor / logic 182 may provide status information about the captured video frames.

[0067] The structured light pattern lens 162 may be a lens for the structured light projector 104. The structured light pattern lens 162 may be configured to enable the emission of the structured light SLP generated by the structured light source 120 of the structured light projector 104 while protecting the structured light source 120. The structured light pattern lens 162 may be configured to decompose the laser pattern generated by the structured light source 120 into a pattern array (e.g., a dense dot pattern array).

[0068] The sensor 164 may implement a plurality of sensors, including but not limited to a motion sensor, an ambient light sensor, a proximity sensor (e.g., ultrasound, radar, lidar, etc.), an audio sensor (e.g., a microphone), etc. In an embodiment implementing a motion sensor, the sensor 164 may be configured to detect motion anywhere in the field of view monitored by the camera system 100. In various embodiments, the detection of motion may be used as a threshold for activating the capture device 106. The sensor 164 may be implemented as an internal component of the camera system 100 and / or as a component external to the camera system 100. In an example, the sensor 164 may be implemented as a passive infrared (PIR) sensor. In another example, the sensor 164 may be implemented as a smart motion sensor. In an embodiment implementing a smart motion sensor, the sensor 164 may include a low-resolution image sensor configured to detect motion and / or a person.

[0069] In various embodiments, the sensor 164 may generate a signal (e.g., SENS). The signal SENS may include various data (or information) collected by the sensor 164. In an example, the signal SENS may include data collected in response to motion being detected in the monitored field of view, ambient light levels in the monitored field of view, and / or sounds picked up in the monitored field of view. However, other types of data may be collected and / or generated based on the design criteria of a particular application. The signal SENS may be presented to the processor / SoC 102. In an example, when motion is detected in the field of view monitored by the camera system 100, the sensor 164 may generate a signal SENS (making the signal SENS effective). In another example, when the sensor 164 is triggered by audio in the field of view monitored by the camera system 100, the sensor 164 may generate a signal SENS (making the signal SENS effective). In yet another example, the sensor 164 may be configured to provide directional information about motion and / or sound detected in the field of view. The directional information may also be communicated to the processor / SoC 102 via the signal SENS.

[0070] HID 166 can implement an input device. For example, HID 166 can be configured to receive human input. In one example, HID 166 can be configured to receive a password input from a user. In some embodiments, camera system 100 may include a keypad, a touchpad (or screen), a doorbell switch, and / or other human interface devices (HIDs) 166. In an example, sensor 164 can be configured to determine when an object approaches HID 166. In an example in which camera system 100 is implemented as part of an access control application, capture device 106 can be turned on to provide an image for identifying a person attempting to access, and lighting of a locked area and / or lighting of a touchpad 166 for access can be turned on. For example, a combination of inputs from HID 166 (e.g., a password or PIN number) can be combined with presence determination and / or depth analysis performed by processor 102 to implement two-factor authentication.

[0071] Processor / SoC 102 may receive signal VIDEO and signal SENS. Processor / SoC 102 may generate one or more video output signals (e.g., VIDOUT) and one or more control signals (e.g., CTRL) based on signal VIDEO, signal SENS, and / or other inputs. In some embodiments, signals VIDOUT and CTRL may be generated based on analysis of signal VIDEO and / or an object detected in signal VIDEO. In various embodiments, processor / SoC 102 may be configured to perform one or more of the following: feature extraction, object detection, object tracking, and object recognition. For example, processor / SoC 102 may determine motion information and / or depth information by analyzing a frame from signal VIDEO and comparing the frame with a previous frame. Comparison may be used to perform digital motion estimation. In some embodiments, processor / SoC 102 may be configured to generate a video output signal VIDOUT including video data from signal VIDEO. Video output signal VIDOUT may be presented to memory 150, communication module 154, and / or wireless interface 156.

[0072] The multi-step method for activating and / or disabling the capture device 106 based on the output of the motion sensor 164 and / or any other power consumption features of the camera system 100 may be implemented to reduce the power consumption of the camera system 100 and extend the operational life of the battery 152. The motion sensor of the sensor 164 may have a low power drain on the battery 152 (e.g., less than 10W). In an example, the motion sensor of the sensor 164 may be configured to remain on (e.g., always active) unless disabled in response to feedback from the processor / SoC 102. The video analysis performed by the processor / SoC 102 may have a relatively large power drain on the battery 152 (e.g., greater than the motion sensor 164). In an example, the processor / SoC 102 may be in a low power state (or powered down) until some motion is detected by the motion sensor of the sensor 164.

[0073] The camera system 100 may be configured to operate using various power states. For example, in a power-down state (e.g., a sleep state, a low-power state), the motion sensor of the sensor 164 and the processor / SoC 102 may be turned on, and other components of the camera system 100 (e.g., the image capture device 106, the memory 150, the communication module 154, etc.) may be turned off. In another example, the camera system 100 may operate in an intermediate state. In the intermediate state, the image capture device 106 may be turned on, and the memory 150 and / or the communication module 154 may be turned off. In yet another example, the camera system 100 may operate in a powered (or high-power) state. In the powered state, the sensor 164, the processor / SoC 102, the capture device 106, the memory 150, and / or the communication module 154 may be turned on. The camera system 100 may consume some power (e.g., a relatively small and / or minimal amount of power) from the battery 152 in the powered-down state. In the powered-on state, the camera system 100 may consume more power from the battery 152. As camera system 100 operates in each of the power states, the power states and / or the number of components of camera system 100 that are turned on may vary according to the design criteria of a particular implementation.

[0074] The processor 102 shown includes a plurality of blocks (or circuits) 190a-190n. Blocks 190a-190n may implement various hardware modules implemented by the processor 102. Hardware modules 190a-190n may be configured to provide various hardware components to implement a video processing pipeline. Circuits 190a-190n may be configured to receive pixel data VIDEO, generate video frames according to pixel data, perform various operations on video frames (e.g., de-warp, rolling shutter correction, cropping, magnification, image stabilization, etc.), prepare video frames for communication with external hardware (e.g., encoding, packaging, color correction, etc.), parse feature sets, implement various operations of computer vision (e.g., object detection, segmentation, classification, etc.), etc. Various implementations of the processor 102 may not necessarily utilize all features of hardware modules 190a-190n. The features and / or functions of hardware modules 190a-190n may vary according to the design criteria of a particular implementation. Details of hardware modules 190a-190n may be described in association with U.S. patent application Ser. No. 16 / 831,549 filed on April 16, 2020, U.S. patent application Ser. No. 16 / 288,922 filed on February 28, 2019, and U.S. patent application Ser. No. 15 / 593,493 filed on May 12, 2017 (now U.S. Patent No. 10,437,600), and U.S. patent application Ser. No. 16 / 991,344 filed on August 12, 2020, appropriate portions of which are incorporated herein by reference in their entirety.

[0075] Hardware modules 190a-190n can be implemented as dedicated hardware modules.Compared with software implementation, using dedicated hardware modules 190a-190n to implement various functions of processor 102 can enable processor 102 to be highly optimized and / or customized to limit power consumption, reduce heat generation and / or increase processing speed.Hardware modules 190a-190n can be customizable and / or programmable to implement various types of operations.Implementation of dedicated hardware modules 190a-190n can enable the hardware used to perform each type of calculation to be optimized for speed and / or efficiency.For example, hardware modules 190a-190n can implement multiple relatively simple operations frequently used in computer vision operations, which can enable computer vision operations to be performed in real time together.Video pipelines can be configured to identify objects.Objects can be identified by interpreting digital and / or symbolic information to determine that visual data represents a specific type and / or feature of an object.For example, the number of pixels of video data and / or the color of pixels can be used to identify parts of video data as objects.

[0076] One of the hardware modules 190a-190n (e.g., 190a) can implement a scheduler circuit. Scheduler circuit 190a can be configured to store a directed acyclic graph (DAG). In an example, scheduler circuit 190a can be configured to generate and store a directed acyclic graph in response to feature set information received (e.g., loaded). A directed acyclic graph can define video operations to be performed to extract data from a video frame. For example, a directed acyclic graph can define various mathematical weights to be applied when performing computer vision operations to classify various pixel groups as specific objects.

[0077] The scheduler circuit 190a may be configured to parse the acyclic graph to generate various operators. Operators may be scheduled by the scheduler circuit 190a in one or more of the other hardware modules 190a-190n. For example, one or more of the hardware modules 190a-190n may implement a hardware engine configured to perform a specific task (e.g., a hardware engine designed to perform specific mathematical operations repeatedly used to perform computer vision operations). The scheduler circuit 190a may schedule operators based on when the operators are ready to be processed by the hardware engines 190a-190n.

[0078] The scheduler circuit 190a can time-multiplex tasks to the hardware modules 190a-190n to perform work based on the availability of the hardware modules 190a-190n. The scheduler circuit 190a can parse the directed acyclic graph into one or more data streams. Each data stream can include one or more operators. Once the directed acyclic graph is parsed, the scheduler circuit 190a can assign the data stream / operator to the hardware engine 190a-190n and send the relevant operator configuration information to start the operator.

[0079] The binary representation of each directed acyclic graph can be an ordered traversal of the directed acyclic graph, where descriptors and operators are interleaved based on data dependencies. Descriptors typically provide registers that link data buffers to specific operands in dependent operators. In various embodiments, an operator may not appear in the directed acyclic graph representation until all dependent descriptors have been declared for the operands.

[0080] One of the hardware modules 190a-190n (e.g., 190b) may implement a convolutional neural network (CNN) module. The CNN module 190b may be configured to perform computer vision operations on video frames. The CNN module 190b may be configured to implement object recognition through multi-layer feature detection. The CNN module 190b may be configured to calculate descriptors based on the feature detection performed. The descriptors may enable the processor 102 to determine the likelihood that a pixel of a video frame corresponds to a specific object (e.g., a specific make / model / year of a vehicle, identifying a person as a specific individual, detecting the type of animal, etc.).

[0081] The CNN module 190b may be configured to implement convolutional neural network capabilities. The CNN module 190b may be configured to implement computer vision using deep learning techniques. The CNN module 190b may be configured to implement pattern and / or image recognition using a training process through multi-layer feature detection. The CNN module 190b may be configured to perform inference for a machine learning model.

[0082] CNN module 190b can be configured to perform feature extraction and / or matching only in hardware. Feature points typically represent areas of interest (e.g., corners, edges, etc.) in a video frame. By temporarily tracking feature points, an estimate of the self-motion of the capture platform or a motion model of an object observed in the scene can be generated. In order to track feature points, a matching algorithm is usually incorporated into CNN module 190b by hardware to find the most likely correspondence between feature points in a reference video frame and a target video frame. In the process of matching reference feature points and target feature points, each feature point can be represented by a descriptor (e.g., image block, SIFT, BRIEF, ORB, FREAK, etc.). Using dedicated hardware circuits to implement CNN module 190b can achieve real-time calculation of descriptor matching distances.

[0083] CNN module 190b may be configured to perform face detection, face recognition, and / or presence determination. For example, face detection, face recognition, and / or presence determination may be performed based on a trained neural network implemented by CNN module 190b. In some embodiments, CNN module 190b may be configured to generate a depth image based on a structured light pattern. CNN module 190b may be configured to perform various detection and / or recognition operations and / or perform 3D recognition operations.

[0084] CNN module 190b can be a dedicated hardware module configured to perform feature detection of video frames. Features detected by CNN module 190b can be used to calculate descriptors. CNN module 190b can determine the possibility that pixels in video frames belong to a specific object and / or multiple objects in response to descriptors. For example, using descriptors, CNN module 190b can determine the possibility that pixels correspond to specific objects (e.g., people, furniture items, pets, vehicles, etc.) and / or features of objects (e.g., the shape of the eyes, the distance between facial features, the hood of a vehicle, body parts, license plates of vehicles, faces of people, clothes worn by people, etc.). Implementing CNN module 190b as a dedicated hardware module of processor 102 can enable device 100 to perform computer vision operations locally (e.g., on-chip) without relying on the processing power of remote devices (e.g., transmitting data to cloud computing services).

[0085] The computer vision operations performed by the CNN module 190b can be configured to perform feature detection on the video frame to generate a descriptor. The CNN module 190b can perform object detection to determine the area in the video frame with a high probability of matching a specific object. In one example, an open operand stack can be used to customize the type of object to be matched (e.g., a reference object) (thereby enabling the programmability of the processor 102 to implement various artificial neural networks defined by directed acyclic graphs, each directed acyclic graph providing instructions for performing various types of object detection). The CNN module 190b can be configured to perform local masking to detect objects in areas with a high probability of matching (multiple) specific objects.

[0086] In some embodiments, CNN module 190b can determine the location (e.g., 3D coordinates and / or position coordinates) of various features of the detected object. In one example, 3D coordinates can be used to determine the position of a person's arms, legs, chest, and / or eyes. A position coordinate for the vertical position of a body part in 3D space on a first axis and another coordinate for the horizontal position of a body part in 3D space on a second axis can be stored. In some embodiments, the distance from lens 160 can represent a coordinate for the depth position of a body part in 3D space (e.g., a position coordinate on a third axis). Using the position of each body part in 3D space, processor 102 can determine body location and / or physical characteristics of the detected person.

[0087] The CNN module 190b may be pre-trained (e.g., the CNN module 190b may be configured to perform computer vision based on received training data to detect an object to train the CNN module 190b). For example, the results of the training data (e.g., a machine learning model) may be pre-programmed and / or loaded into the processor 102. The CNN module 190b may perform inferences for the machine learning model (e.g., to perform object detection). Training may include determining weight values ​​for each layer of the neural network model. For example, weight values ​​may be determined for each of the layers for feature extraction (e.g., convolutional layers) and / or the layers for classification (e.g., fully connected layers). The weight values ​​learned by the CNN module 190b may vary according to the design criteria of a particular implementation.

[0088] The convolution operation may include sliding a feature detection window along the layer while performing calculations (e.g., matrix operations). The feature detection window may apply filters to pixels and / or extract features associated with each layer. The feature detection window may be applied to a pixel and a plurality of surrounding pixels. In an example, a layer may be represented as a matrix representing the values ​​of a pixel and / or a feature of one of the layers, and the filter applied by the feature detection window may be represented as a matrix. The convolution operation may apply matrix multiplication between regions of the current layer covered by the feature detection window. The convolution operation may slide the feature detection window along regions of the layer to generate a result representing each region. The size of the region, the type of operation applied by the filter, and / or the number of layers may vary according to the design criteria of a particular implementation.

[0089] Using convolution operations, CNN module 190b can calculate multiple features for pixels of the input image in each extraction step. For example, each of the layers can receive input from a feature set in a small neighborhood (e.g., region) located in a previous layer (e.g., local receptive field). Convolution operations can extract basic visual features (e.g., oriented edges, endpoints, corners, etc.), which are then combined by higher layers. Since the feature extraction window operates on pixels and nearby pixels, the result of the operation can have position invariance. The layer may include a convolution layer, a pooling layer, a nonlinear layer, and / or a fully connected layer. In an example, a convolution operation can learn to detect edges from raw pixels (e.g., the first layer), then use features from the previous layer (e.g., detected edges) to detect shapes in the next layer, and then use the shapes to detect higher-level features in the higher layer (e.g., facial features, pets, vehicles, components of vehicles, furniture, etc.), and the last layer can be a classifier using higher-level features.

[0090] CNN module 190b can execute data flows directed to feature extraction and matching, including two-stage detection, deformation operators, component operators that manipulate lists of components (e.g., components can be regions of vectors that share common attributes and can be grouped together with bounding boxes), matrix inversion operators, dot product operators, convolution operators, conditional operators (e.g., multiplexing and demultiplexing), remapping operators, minimum and maximum reduction operators, pooling operators, non-minimum non-maximum suppression operators, non-maximum suppression operators based on scanning windows, aggregation operators, scattering operators, statistical operators, classifier operators, integral image operators, comparison operators, indexing operators, pattern matching operators, feature extraction operators, feature detection operators, two-stage object detection operators, score generation operators, block reduction operators, and upsampling operators. The types of operations performed by CNN module 190b to extract features from training data can vary according to the design criteria of a particular implementation.

[0091] Each of the hardware modules 190a-190n may implement a processing resource (or hardware resource or hardware engine). The hardware engines 190a-190n may be operable to perform specific processing tasks. In some configurations, the hardware engines 190a-190n may operate in parallel and independently of each other. In other configurations, the hardware engines 190a-190n may operate in conjunction with each other to perform assigned tasks. One or more of the hardware engines 190a-190n may be a homogeneous processing resource (all circuits 190a-190n may have the same capabilities) or a heterogeneous processing resource (two or more circuits 190a-190n may have different capabilities).

[0092] refer to Figure 3 , a diagram illustrating a rolling shutter system in a camera system including an RGB-IR rolling shutter sensor is shown. An example embodiment 200 is shown. The example embodiment 200 may include an embodiment of the camera system 100. The camera system 100 may include a structured light projector 104 and / or a capture device 106.

[0093] The example embodiment 200 may be an example of a camera system 100 implementing an RGB-IR rolling shutter sensor. The RGB-IR rolling shutter sensor 130 is shown as a component of the capture device 106. In one example, a dynamic driver mechanism for a rolling shutter sensor implemented by the camera system 100 may be configured to operate with the RGB-IR rolling shutter sensor 130.

[0094] Camera system 100 is shown generating pixel data 202 and pixel data 204. Pixel data 202 may be pixel data that may be used to generate an IR image (or video frame). Pixel data 204 may be pixel data that may be used to generate an RGB image (or video frame). A video processing pipeline (e.g., ISP circuit 108) of processor 102 may be configured to generate an IR image in response to IR pixel data 202 in signal VIDEO generated by RGB-IR rolling shutter sensor 130. A video processing pipeline of processor 102 may be configured to generate an RGB image in response to RGB pixel data 204 in signal VIDEO generated by RGB-IR rolling shutter sensor 130. In the example shown, IR pixel data 202 and RGB pixel data 204 are shown as being external to camera system 100. However, in general, IR pixel data 202 (and the resulting IR image) and / or RGB pixel data 204 (and the resulting RGB image) may be used internally by processor 102 (eg, signal IR DATA and signal RBG DATA received by control circuitry 112).

[0095] refer to Figure 4, a diagram illustrating a rolling shutter system in a camera system including separate IR rolling shutter sensors and RGB rolling shutter sensors is shown. An example embodiment 200' is shown. The example embodiment 200' may include an embodiment of the camera system 100'. The camera system 100' may include a structured light projector 104 and / or a capture device 106.

[0096] Example embodiment 200' can be an example of a camera system 100' implementing a separate IR rolling shutter sensor and an RGB rolling shutter sensor. Capture device 106 is shown as including IR rolling shutter sensor 130a and RGB rolling shutter sensor 130b. Lenses 160a-160b are shown. Lens 160a can be a lens for IR rolling shutter sensor 130a, and lens 160b can be a lens for RGB rolling shutter sensor 130b. In one example, a dynamic actuator mechanism for a rolling shutter sensor implemented by camera system 100' can be configured to operate with IR rolling shutter sensor 130a and RGB rolling shutter sensor 130b.

[0097] The camera system 100' is shown generating IR pixel data 202 and RGB pixel data 204. An IR image may be generated in response to the IR pixel data 202 generated by the IR rolling shutter sensor 130a. An RGB image may be generated in response to the RGB pixel data 204 generated by the RGB rolling shutter sensor 130b. The video processing pipeline of the processor 102 may be configured to generate an IR image in response to the IR pixel data 202 generated by the IR rolling shutter sensor 130a. The video processing pipeline of the processor 102 may be configured to generate an RGB image in response to the RGB pixel data 204 generated by the RGB rolling shutter sensor 130b. In the example shown, the IR pixel data 202 and the RGB pixel data 204 are shown as being external to the camera system 100'. However, in general, IR pixel data 202 (and IR image) and / or RGB pixel data 204 (and RGB image) may be used internally by processor 102 (eg, signal IR DATA and signal RBG DATA received by control circuitry 112).

[0098] refer to Figure 5 , a diagram illustrating elements of a 3D sensing system according to an example embodiment of the present invention is shown. A 3D sensing system 250 is shown. The 3D sensing system 250 may include the camera system 100. In the example shown, the 3D sensing system 250 may be implemented with Figure 3RGB-IR camera system 100 is shown in association. In some embodiments, 3D sensing system 250 can implement camera system 100 having IR sensor 130a separate from RGB sensor 130b. Camera system 100 can be implemented in a housing (e.g., a sealed housing, a weatherproof housing, a protective housing, etc.). For example, infrared (IR) structured light projector 104 (including structured light pattern lens 162 and structured light source 120) and camera 106 (including lens 160 and a single RGB-IR image sensor 130) can be mounted in housing 202.

[0099] The IR structured light projector 104 can be configured to project the structured light pattern SLP onto an object in the field of view of the capture device 106 when turned on. The RGB-IR sensor 130 can be used to acquire both IR image data (with and without the structured light pattern SLP) and RGB image data for objects in the field of view of the capture device 106. The 3D sensing system 200 generally provides advantages over conventional two camera sensing systems. For example, by utilizing only one RGB-IR rolling shutter sensor 130 to acquire both RGB image data and IR image data with / without the structured light pattern SLP, the 3D sensing system 200 generally reduces system cost and system complexity relative to conventional systems using global shutter sensors (e.g., one sensor and one lens versus two sensors and two lenses, and rolling shutter versus global shutter).

[0100] The structured light pattern SLP is shown as a dashed line. The dashed line may indicate that the structured light pattern SLP may not always be present (e.g., only present when the structured light projector 104 is activated). When the projector 104 is activated, the processor 102 may receive data corresponding to the structured light pattern SLP as part of the IR data 202. When the projector 104 is not activated, the processor 102 may not receive data corresponding to the structured light pattern SLP. IR pixel data 202 generated by the RGB-IR rolling shutter sensor 130 may be received by the processor 102. RGB pixel data 204 generated by the RGB-IR rolling shutter data 130 may be received by the processor 102. For example, the processor 102 may receive the IR pixel data 202 and the RGB pixel data 204 without the structured light pattern SLP, or receive the IR pixel data 202 and the RGB pixel data 204 with the structured light pattern SLP.

[0101] The video processing pipeline (e.g., ISP circuit 108) of processor 102 (e.g., which implements one or more of hardware modules 190a-190n) can be configured to generate an image and / or video frame in response to IR pixel data 202, RGB pixel data 204, and / or structured light pattern SLP. In an example, processor 102 can be configured to separate (divide) IR pixel data 202 and RGB pixel data 204 from RGB-IR sensor 130 into an IR image data channel (e.g., IR DATA) and an RGB image data channel (e.g., RGB DATA). Processor 102 can be configured to generate a video frame based on IR pixel data 202 and / or RGB pixel data 204.

[0102] In an example, the IR image data channel and / or the RGB image data channel may be processed by the processor 102 for 3D (e.g., depth) perception, presence determination, object detection, face detection, object identification, and face recognition. In an example, the IR image data channel with the structured light pattern SLP present may be used to perform depth analysis and presence determination. The IR image data and RGB image data channels without the structured light pattern SLP may be used to perform object detection, face detection, object identification, and face recognition.

[0103] In an example, when the structured light pattern SLP is present (e.g., the structured light projector 104 is on or in response to the signal SLTRIG), the depth analysis 252 can be applied to the IR image data channel. The depth analysis 252 can also be utilized to make presence determinations. In an example, the depth analysis 252 can be utilized to distinguish between image frames that include real people and image frames that include representations of people (e.g., photos, images displayed on a computer, images displayed on a smart phone, etc.). In an example, both the IR image data channel and the RGB image data channel when the structured light pattern SLP is not present (e.g., the structured light projector 104 is off) can be used to perform object detection, facial detection, object identification, facial recognition, etc.

[0104] In an example, the IR image data channel and / or the RGB image data channel when the structured light pattern SLP is not present (e.g., the structured light projector 104 is off) can be presented as input to the hardware modules 190a-190n. In an example, one or more of the hardware modules 190a-190n can be configured to implement one or more neural network architectures, including but not limited to convolutional neural networks (CNNs), deep learning neural networks (DNNs), fully connected neural networks, etc. A neural network (more properly referred to as an "artificial" neural network (ANN)) is a computing system composed of multiple simple, highly interconnected processing elements (e.g., implemented by one or more of the hardware modules 190a-190n) that process information by responding to external inputs with corresponding dynamic states. A neural network is a processing device (algorithm or actual hardware) that loosely mimics the neuronal structure of the brain but on a much smaller scale. Large neural networks can have hundreds or thousands of processing units.

[0105] Neural networks are typically organized in layers. The layers may include multiple interconnected nodes that perform operations or activation functions. Input data is presented to an input layer of the network, which communicates with one or more internal (or hidden) layers where the actual processing is performed via an arrangement of weighted connections. The process of determining the weights for the weighted connections of the neural network is typically referred to as training. The hidden layers are connected to an output layer where the results of the predictive model implemented by the neural network are presented.

[0106] In an example, the CNN module 190b can be configured (e.g., by design, by a training process, etc.) to obtain detection, recognition, and / or classification results in real time. The CNN module 190b can be configured to perform various computer vision operations 254. In an example, in a low-light (or night) environment, the computer vision operation 254 can utilize the IR image data channel to identify features of people in the generated image and / or identify inanimate objects (e.g., computers, vehicles, trees, etc.) in the image. In a brighter (daytime) environment, the computer vision operation 254 can utilize the RGB image data channel to make inferences based on the color image (e.g., to detect people and / or inanimate objects, etc.).

[0107] The depth analysis 252 and / or computer vision operations 254 may be configured to generate results 256. The processor 102 may be configured to utilize the results of the depth analysis 252 and computer vision operations 254, alone or in combination. The results 256 may include data corresponding to the identification, detection, and / or location of an object in a video frame. In an example, the results 256 may include a depth image.

[0108] IR data with structured light pattern SLP can be used to perform depth analysis 252, and then the result 256 of depth analysis 252 can be used for presence determination. RGB data 204 and / or IR data 202 without structured light pattern SLP can be used to perform computer vision operations 254 (e.g., for face detection, object detection, etc.). The combination of depth analysis 252 and computer vision operations 254 can generate results 256, which can include presence results and facial recognition results. The result of facial detection can be (multiple) coordinates of the face. The coordinates of the face can be used to crop data from the original image for facial recognition. The coordinates of the face can also be used to control activation (e.g., dynamically turn on the structured light projector 104), and then the received IR data 202 with SLP will be used to perform depth analysis 252 on the facial ROI.

[0109] For example, processor 102 may use result 256 to determine the location of an object of interest in a video frame (e.g., X, Y coordinates in pixels of a video frame, the location of a bounding box corresponding to the object of interest in the video frame, etc.). Result 256 may include information about the classification of the detected object (e.g., the type of object detected, whether the detected object matches a reference object, etc.). Result 256 may be used by processor 102 to perform facial detection and / or recognition (e.g., to identify a detected person as a particular person). Utilization of result 256 by processor 102 may vary depending on the design criteria of a particular implementation.

[0110] refer to Figure 6 , a diagram illustrating the start and end lines of detecting a region of interest in an image frame is shown. An image frame (or video frame) 300 is shown. Image frame 300 may be a representative example of a video frame generated by processor 102. In an example, video frame 300 may be used internally by processor 102 to perform computer vision operations 254. In another example, video frame 300 may be presented as an output of camera system 100 (e.g., as signal VIDOUT). For example, video frame 300 may be presented to a display system.

[0111] In the example shown, the video frame 300 may represent a video frame generated in response to the absence of IR image data (e.g., from the IR pixel data 202) and RGB image data channels (e.g., from the RGB pixel data 204) in the structured light pattern output SLP. For example, the processor 102 may use the video frame 300 to perform object detection, face detection, and / or determine the location of an object in the video frame 300. The example video frame 300 may include an image of a person 302. The person 302 is shown in a portion of the video frame 300 (e.g., the video data corresponding to the person 302 does not occupy the entire video frame). The person 302 may be an object of interest. In general, a video frame may include video data corresponding to other objects and / or scenery (e.g., background, trees, ground, sky, distant objects, etc.). For illustrative purposes, the example video frame 300 may only show the person 302.

[0112] A dashed box 304 is shown. Dashed box 304 may correspond to the face of person 302. For example, the face of person 302 may be an object of interest. For example, video frame 300 may be an example of camera system 100 used in a facial recognition application. Dashed box 304 may represent a bounding box of a detected object. Bounding box 304 may be shown for illustrative purposes. Typically, bounding box 304 may not be shown in output video signal VIDOUT when output to a display (e.g., bounding box 304 may be shown in a debug mode of operation to indicate which objects have been detected in a video frame and / or to ensure that facial recognition is being performed). Bounding box 304 may represent a location (e.g., coordinate values) in video frame 300 where an object of interest has been detected.

[0113] The object of interest may be detected by the processor 102. The specific object of interest detected may be determined in response to computer-readable instructions and / or reference objects stored by the memory 150. The processor 102 may be configured to detect various types of objects of interest. The type of object of interest may be defined by an artificial neural network implemented by the processor 102. In the example shown, the artificial neural network implemented by the CNN module 190b may be configured to detect a human face. In the example shown, the object of interest may be the face of the person 302. When the camera system 100 is used to detect a person and / or perform facial recognition, the face of the person may be the object of interest. In some embodiments, the object of interest may be an animal and / or a specific type of animal (e.g., when the camera system 100 is used as a trail camera for natural video / images). In some embodiments, the object of interest may be a vehicle and / or a specific make / model / year of a vehicle (e.g., when the camera system 100 is implemented as part of a vehicle camera system and / or a traffic camera). The type of object of interest detected may be an option selectable by the end user. In some embodiments, the type of object of interest may be defined in a directed acyclic graph loaded by the scheduler circuit 190a. In the example shown, one object of interest is detected. In some embodiments, processor 102 may be configured to detect more than one object of interest.The number and / or type of objects of interest detected in a video frame by processor 102 may vary according to the design criteria of a particular implementation.

[0114] Line 310 and line 312 are shown. Line 310 and line 312 may each include a row of pixel data of video frame 300. Line 310 may include a start line position corresponding to detected object 304 (e.g., object of interest). Line 312 may include an end line position corresponding to detected object 304 (e.g., object of interest). In the example shown, start line position 310 may generally correspond to the top of the face of person 302 and / or the top of bounding box 304. In the example shown, end line position 312 may generally correspond to the bottom of the face of person 302 and / or the bottom of bounding box 304. Start line position 310 may correspond to an exposure time of RGB-IR rolling shutter sensor 130. End line position 312 may correspond to an exposure time of RGB-IR rolling shutter sensor 130.

[0115] The processor 102 may be configured to control activation of the IR light projector 104 using the start line position 310 and the end line position 312. The processor 102 may be configured to dynamically control activation of the structured light pattern SLP to align with the timing of the start line position 310 and the end line position 312. Aligning the timing of activation of the structured light pattern SLP with the start line position 310 and the end line position 312 may ensure that the structured light pattern SLP is active when the RGB-IR rolling shutter sensor 130 is currently exposing a portion of the sensor that corresponds to the location of the detected object 304.

[0116] In some embodiments, the object of interest may be a general classification of objects. For the purpose of determining the start line location 310 and the end line location 312, the computer vision operation 254 may not need to perform fine-grained classification of the object 304. For example, when the camera system 100 is configured to perform facial recognition, the processor 102 may be configured to first detect a human face to determine the start line location 310 and the end line location 312. For the purpose of determining the start line location 310 and the end line location 312, the processor 102 may not need to identify the detected face as the face of a specific person. For example, when an image is captured with the structured light pattern SLP activated, further analysis performed using the depth analysis 252 may be performed.

[0117] The processor 102 may be configured to dynamically locate the object of interest 304 (e.g., track its position) over time. For example, the position of the object of interest 304 may change over time (e.g., the person 302 may move, walk closer to the lens 160, move away from the lens 160, etc.). The processor 102 may continuously and / or continuously detect the object of interest 304 and / or determine the position of the object of interest 304. As the position of the object of interest 304 changes, the processor 102 may update the start line position 310 and the end line position 312. By updating the start line position 310 and the end line position 312 as the object of interest 304 moves (e.g., the position changes over time, the position changes in the sequence of video frames, etc.), the processor 102 may enable the activation of the structured light source 120 to be dynamically adjusted. Dynamically adjusting the activation of the structured light source 120 may ensure that the structured light pattern SLP is synchronized with the rolling shutter sensor 130 and the position of the object of interest 304.

[0118] refer to Figure 7, a diagram illustrating the start and end times of exposure for an IR image frame is shown. An IR image frame 350 is shown. The IR image frame 350 may be a representative example of an IR frame generated by the processor 102 in response to the IR pixel data 202. The example IR image frame 350 may include image data generated in response to the IR data channel when the structured light pattern SLP is active. The structured light pattern SLP may be activated only for a subset of the amount of time that the RGB-IR rolling shutter is used to generate the IR pixel data 202 (which is used to generate the IR image frame 350). In an example, the IR frame 350 may be used internally by the processor 102 to perform the depth sensing operation 252.

[0119] Line 352 and line 354 are shown. Line 352 may correspond to Figure 6 The start line position 310 is shown in association with FIG. 354. Figure 6 The start line position 352 and the end line position 354 may be used to calculate the timing of activation and / or deactivation of the structured light projector 104 .

[0120] IR image frame 350 may include portion 360, portion 362, and portion 364. Portion 360 may generally be a top portion of IR image frame 350. Portion 360 may be a portion of IR image frame 350 located above start line location 352. Portion 362 may generally include a middle portion of IR image frame 350. Portion 362 may be a portion of IR image frame 350 located between start line location 352 and end line location 354. Portion 364 may generally be a bottom portion of IR image frame 350. Portion 364 may be a portion of IR image frame 350 below bottom line location 354. The location and / or size of portion 360, portion 362, and / or portion 364 may depend on the location of start line location 352 and end line location 354.

[0121] Portion 362 may be the location of a region of interest. Region of interest 362 may correspond to Figure 6The position of the object of interest determined from the video frame 300 is shown in association. In some embodiments, if the object of interest 304 is at the top of the IR image frame 350 (e.g., the top of the bounding box 304 is the top line of the IR image frame 350), the region of interest 362 may start at the top of the IR image frame 350 (e.g., portion 360 may not exist, and only portion 364 may be below the region of interest 362). In some embodiments, if the object of interest 304 is at the bottom of the IR image frame 350 (e.g., the bottom of the bounding box 304 is at the bottom of the IR image frame 350), the region of interest 362 may end at the bottom of the IR image frame 350 (e.g., portion 364 may not exist, and only portion 360 may be above the region of interest 362).

[0122] RGB-IR rolling shutter sensor 130 may be configured to capture IR pixel data 202 by implementing row-by-row exposure from top to bottom to capture IR image frame 350. In the example shown, top 360 may be captured first, then region of interest 362, and then bottom 364. When RGB-IR rolling shutter sensor 130 is acquiring IR pixel data 202 and / or RGB pixel data 204 to generate an IR image frame 350, the IR pixel data 202 may be captured by the RGB-IR rolling shutter sensor 130. Figure 6 The structured light projector 104 may be turned off (e.g., deactivated) when the video frame 300 is shown in association. The RGB-IR rolling shutter sensor 130 may be sequentially exposed row by row over time to generate the IR image 350. When the RGB-IR rolling shutter sensor 130 captures the top portion 360, the structured light source 120 may remain deactivated.

[0123] When the RGB-IR rolling shutter sensor 130 reaches the start line position 352 (or arrives slightly earlier depending on the activation time and / or the desired integration amount), the SL control circuit 112 can generate a signal SL TRIG. The signal SL TRIG can activate the structured light source 120 (e.g., when the structured light source 120 is in a deactivated state). For example, the structured light source 120 can generate a structured light pattern SLP in response to the signal SL TRIG. The RGB-IR rolling shutter sensor 130 can continue to be exposed row by row sequentially over time. When the RGB-IR rolling shutter sensor captures the region of interest portion 362, the structured light source 120 can provide the structured light pattern SLP (e.g., operating in an activated state).

[0124] When the RGB-IR rolling shutter sensor 130 reaches the end line position 354 (or arrives slightly later depending on the deactivation time and / or the desired integration amount), the SL control circuit 112 can generate a signal SL TRIG. The signal SL TRIG can deactivate the structured light source 120 (e.g., when the structured light source 120 is in an activated state). For example, the structured light source 120 can generate a structured light pattern SLP until the signal SL TRIG is received (e.g., until the end line position 354 is reached). The RGB-IR rolling shutter sensor 130 can continue to be exposed row by row sequentially over time. The RGB-IR rolling shutter sensor 130 can capture the bottom area 364 with the structured light source 120 in an operational deactivated state. After all rows of the RGB-IR rolling shutter sensor 130 have been exposed, the RGB-IR rolling shutter sensor 130 can repeat the exposure process by sequentially exposing from the top row to the bottom row.

[0125] If the object of interest 304 moves, the start line position 352 and the end line position 354 of the timing of activation and / or deactivation of the structured light pattern SLP can be dynamically adjusted. By controlling the activation / deactivation of the structured light pattern SLP according to the position of the object of interest 304 in the video frame, the structured light pattern SLP can be captured by the rolling shutter sensor 130 at a specified frame without affecting the quality of images in other video frames.

[0126] The image signal processor 108 may be configured to generate an interrupt signal ISP SYNC and / or a signal EOF. The circuit 112 may be configured to generate a signal SL TRIG in response to the interrupt signal ISP SYNC and / or the interrupt signal EOF. The signal ISPSYNC may be configured to synchronize the timing of the structured light source 120 with the timing of the exposure of the rolling shutter sensor 130. The interrupt signal ISP SYNC and / or the interrupt signal EOF may be generated by the image signal processor 108 in response to data in the signal VIDEO generated by the rolling shutter sensor 130. In one example, the signal VIDEO may include a start line position 310, and the signal ISP SYNC and / or the interrupt signal EOF may provide data about the start line position 310 to the circuit 110. In another example, the signal VIDEO may include an end line position 312, and the signal ISP SYNC and / or the interrupt signal EOF may provide data about the end line position 312 to the circuit 110. In yet another example, the signal VIDEO may include other interrupts with a consistent frame rate, and the signal ISP SYNC and / or the interrupt signal EOF may provide data about the frame rate to the circuit 110. The circuit 110 may use data from the interrupt signal ISP SYNC and / or the interrupt signal EOF to calculate when the rolling shutter sensor 130 may start exposing the first row of positions 352. When the processor 102 determines when the rolling shutter sensor 130 may start exposing the first row of positions 352, the circuit 110 may calculate the turn-on time of the structured light projector 104 based on the position of the object of interest 304.

[0127] The processor 102 may be configured to perform an image cropping operation on the IR image frame 350. The processor 102 may be configured to crop a region of interest 362 from the IR image frame 350. The region of interest 362 may include a structured light pattern SLP captured with a desired timing and / or visual effect for performing a depth analysis 252 on the object of interest 304. The top portion 360 and the bottom portion 364 may not include a structured light pattern SLP captured with a desired timing and / or visual effect for performing a depth analysis 252. By cropping the region of interest 362, only visual data with the structured light pattern SLP may be used for the depth analysis 252, while other portions 360 and 364 that do not have appropriate data may be discarded.

[0128] refer to Figure 8 , a diagram illustrating a timing diagram for sensor exposure is shown. A timing diagram 400 is shown. Timing diagram 400 may provide an illustrative example of exposure timing of rolling shutter sensor 130 and / or timing of activation of structured light projector 104.

[0129] The timing diagram 400 may include a horizontal axis 402. The horizontal axis 402 may represent time. For example, the left side of the horizontal axis 402 may be a time that occurs earlier than the right side of the horizontal axis 402.

[0130] Lines 404a-404b are shown extending diagonally from horizontal axis 402. Line 404a may generally represent a sensor / shutter exposure start time for rolling shutter sensor 130. Line 404b may generally represent a sensor / shutter exposure end time for rolling shutter sensor 130. A horizontal line with a double-headed arrow 406 is shown between lines 404a-404b. Horizontal arrow 406 may be an amount of time for an exposure of rolling shutter sensor 130.

[0131] Horizontal bars 408a-408n are shown between lines 404a-404b. Each of the horizontal bars 408a-408n may represent the exposure of a row of pixels of the rolling shutter sensor 130. For example, horizontal bar 408a may represent the top row of pixels of the rolling shutter sensor 130, horizontal bar 408b may represent the second row of pixels of the rolling shutter sensor 130, and horizontal bar 408n may represent the bottom row of pixels of the rolling shutter sensor 130. Each of the rows 408a-408n of pixels of the rolling shutter sensor 130 may have the same amount of exposure time 406.

[0132] Line intersections 410a-410n and line intersections 412a-412n are shown. Line intersections 410a-410n may be intersections between line 404a and each of horizontal bars 408a-408n. Line intersections 410a-410n may each represent a start exposure time for each of rows of pixels 408a-408n. Line intersections 412a-412n may be intersections between line 404b and each of horizontal bars 408a-408n. Line intersections 412a-412n may each represent an end exposure time for each of rows of pixels 408a-408n. In the example shown, the exposure start time 410a and end time 412a of the top row 408a may be earlier than the exposure start time 410b and exposure end time 412b of the second row 408b. The exposure start times 410a-410n and the exposure end times 412a-412n may occur sequentially (e.g., the exposure start time 410a of the top row may occur earliest and the exposure end time 412a may occur earliest, and then the exposure end time 412b may occur, and the exposure end time 412n of the last row 408n of pixels may occur last, compared to the exposure start time 410a-410n of the next row).

[0133] Since the exposure time 406 can be the same for each of the rows of pixels 408a-408n, the temporal relationship between the start time 410a-410n and the end time 412a-412n can be the same for each of the rows of pixels 408a-408n. The next row of pixels 408a-408n can begin exposure before the previous row of pixels 408a-408n ends exposure. In the example shown, the start time 410a of the first row of pixels can be earlier than the start time 410b of the second row of pixels 408b. However, the start time 410b of the second row of pixels 408b can occur earlier than the end time 412a of the first row of pixels 408a.

[0134] The staggered start times 410a-410n of the exposure of each 408a-408n in the row of pixels by the rolling shutter sensor 130 may result in visual artifacts (e.g., rolling shutter artifacts such as distortion of moving objects, motion blur, etc.). The video processing pipeline of the processor 102 may be configured to correct for visual artifacts caused by the rolling shutter sensor 130. The staggered start times 410a-410n may also result in the structured light pattern SLP appearing unevenly across each row 408a-408n of pixels if the processor 102 does not synchronize the activation and / or deactivation of the structured light source 120 with the rolling shutter sensor 130.

[0135] A dashed circle 414 is shown. The dashed circle 414 may represent the start line location of the object of interest 304 (e.g., Figure 6 The start line position 310 shown in association with Figure 7 4. In the example shown, a starting line position 414 may be selected for row 408c of pixels.

[0136] A dashed circle 416 is shown. The dashed circle 416 may represent the end line location of the object of interest 304 (e.g., Figure 6 The end line position 312 shown in association with Figure 7 4. In the example shown, the start line position 416 may be located at the row 408g of pixels. For example, the object of interest 304 may be located between the row 408d of pixels and the row 408f of pixels.

[0137] Vertical lines 420a-420b are shown. Vertical line 420a can represent the time of activation of the structured light source 120. Vertical line 420b can represent the time of deactivation of the structured light source 120. Activation time 420a can be aligned with row 408d of pixels. Deactivation time 420b can be aligned with the start time (e.g., 410h) of the end line position 416 of row 408h of pixels.

[0138] For example, to select the ROI 362 including rows 408d to 408f, the processor 102 may select row 408c as the first row with the structured light pattern SLP. The activation time 420a may be selected to start before the end exposure time 412c of the row 408c of pixels. Then, the next row after (e.g., the row 408d of pixels) may be the first row with the structured light pattern SLP with sufficient integration time. Similarly, the deactivation time 420b may be selected to turn off the structured light pattern SLP before the end of the row 408g of pixels and before the start time 410h of the row 408h of pixels. The row before the row 408g of pixels (e.g., the row 408f of pixels) may be the last row with the structured light pattern with sufficient integration time. Additional rows of pixels before or after the ROI 362 (e.g., two rows, three rows, etc.) may be selected to ensure sufficient integration time. Selecting activation time 420a during row 408c of pixels (e.g., a row earlier than the start of ROI 362) and selecting deactivation time 420b during row 408g of pixels (e.g., a row later than the end of ROI 362) can ensure that all 408a-408n of the rows of pixels can have structured light pattern SLP. The number of rows of pixels for activation time 420a before start line position 352 and the number of rows of pixels for deactivation time 420b after end line position 354 can vary according to design criteria of a particular implementation.

[0139] In the example shown, the activation time 420a may occur in the middle of the exposure time of some of the rows of pixels (e.g., during the exposure time of the row 408c of pixels). For example, the row 408c of pixels may not have stopped exposure and may receive the structured light pattern SLP. In the example shown, the activation time 420a may occur after the end exposure time 412b for the row 408b of pixels. Since the activation time 420a may occur after the end exposure time 412a and the end exposure time 412b, the rows 408a-408b of pixels may not capture the structured light pattern SLP.

[0140] Similarly, the rows of pixels after the deactivation time 420b (e.g., the last rows of pixels 408h-408n) may not capture the structured light pattern SLP. The processor 102 may be configured to crop out the image data except for the rows of pixels 408d-408f that include the object of interest 304. For example, the rows of pixels 408d-408f may be retained for the depth analysis 252, and the other rows of pixels 408a-408c and 408g-408n may be discarded.

[0141] The rolling shutter sensor 130 may accumulate photons (integrate) of the structured light pattern SLP during the exposure time of the row 408c of pixels. If the end time 412c of the row 408c of pixels is earlier than the deactivation time 420b of the structured light source 120, the intensity of the structured light pattern SLP for the row 408c of pixels may not be as strong as the intensity of the structured light pattern SLP for the row 408d of pixels. The intensity of the structured light pattern SLP may depend on the integration time. Since the exposure end time 412c is earlier than the deactivation time 420b of the structured light projector 104, the row 408c of pixels may have a shorter integration time than the row 408d of pixels. The structured light pattern SLP may start from the third row (e.g., the row 408c of pixels) until the row 408h of pixels, which starts to be exposed after the end line position 416.

[0142] If the processor 102 determines that the position of the object of interest 304 (e.g., a face) begins at row 408d of pixels, the structured light pattern SLP may begin at the end of row 408c, then all rows after row 408c of pixels (e.g., rows 408d-408g of pixels) may receive the structured light pattern SLP, and all rows before 408c may not receive the structured light pattern SLP because rows 408a-408b have stopped exposing.

[0143] In some embodiments, the exposure time 406 may be reduced so that when the structured light projector 104 is turned on, the row 408 c of pixels will have stopped being exposed (e.g., ensuring that the end exposure time 412 c for the row 408 c of pixels is earlier than the activation time 420 a of the structured light projector 104 ). If the activation time 420 a is after the end time 412 c, then no structured light pattern will be displayed in the row 408 c of pixels.

[0144] Before the end time 412c, a dashed circle 414 is shown at the intersection of the structured light pattern activation line 420a and the row 408c of pixels. For example, the dashed circle 414 can represent the true start line of the structured light pattern SLP (e.g., generally equivalent to Figure 6 The start line position 310 shown in association with Figure 7 Since the integration time for the first few lines may be less than for the region of interest (e.g., Figure 7362) and thus the start line position 352 and / or the end line position 354 may be extended to make the ROI 362 larger than the actual area with the object of interest 304. By capturing a larger region of interest with the activated structured light pattern SLP, the relevant data may be cropped and the structured light pattern SLP may be ignored for portions thereof that may lack clarity (e.g., the first few lines of the start position 352 and the first few lines of the end position 354). Since the integration time of the start line and / or the end line may be smaller than the rest of the ROI 362, the captured image may be too dark. For example, the upper and lower boundaries of the ROI 362 may not be very clear (e.g., dark-normal-dark).

[0145] Controlling the timing of activation of the structured light projector 104 can provide power savings. When the object of interest 304 is not present, the structured light pattern SLP can be turned off. By limiting the activation of the structured light projector 104 to the time range of the exposure of the row 408a-408n of pixels where the object of interest 304 is actually located, the structured light projector 104 can always be deactivated. Turning off the structured light projector 104 can reduce power consumption. Since the structured light pattern SLP may not be needed for the depth analysis 252 outside the ROI 362, the structured light projector 104 can be deactivated without negatively affecting the ability of the depth analysis 252.

[0146] refer to Fig. 9 , a diagram illustrating an example security panel providing a facial disarming feature is shown. In an example, a low-cost 3D sensing system according to an example embodiment of the present invention may be implemented as part of a security panel 450. In an example, the security control panel 450 may be configured to restrict access and / or permissions based on facial recognition. In an example, the 3D information determined using the depth analysis 252 may be used for 3D modeling and presence determination, and may be used to identify a specific user from a face and automatically provide hands-free access (e.g., unlock a door, disarm an alarm system, etc.). In an example, the 3D modeling and presence determination of the depth analysis 252 may be used to identify a specific user and allow the alarm system to reprogram privileges based on facial identification (e.g., only person X or Y may be allowed to change access levels or policies, add users, etc., even if the correct password is entered).

[0147] refer to Fig.10, a diagram illustrating an example access control panel is shown. In various embodiments, a low-cost structured light-based 3D sensing system including the device 100 can be implemented as part of the access control panel 480. In an example, the 3D information generated by capturing the structured light pattern SLP using a dynamic driver mechanism for the rolling shutter sensor 130 can be used for 3D modeling and existence determination. In an example, the low-cost structured light-based 3D sensing system can be used to gain access to payment systems (e.g., ATMs, etc.), disarm alarm systems, and / or allow "tripwire" access to restricted areas (e.g., gardens, garages, houses, etc.). In one example, the low-cost structured light-based 3D sensing system can be configured to identify gardeners / pool maintenance personnel and prohibit the triggering of alarms. In another example, the low-cost structured light-based 3D sensing system can be configured to restrict access to certain times and days of the week. In another example, a low-cost structured light-based 3D sensing system may be configured to allow the alarm system to reprogram privileges based on video / audio identification via access control panel 400 (e.g., only allowing person X or Y to change access levels or policies, add users, etc., even if the correct password is entered).

[0148] refer to Fig.11 , a diagram illustrating an example access control panel using a password is shown. In various embodiments, a low-cost structured light-based 3D sensing system including device 100 may be implemented as part of a secure keypad 500. In an example, the secure keypad 500 may be configured to restrict access and / or permissions based on facial recognition determined using depth analysis 252. In an example, 3D information from depth analysis 252 may be used for 3D modeling and presence determination, and may be used to identify a specific user and allow access, alarm system disarming privileges, alarm system programming privileges, and / or alarm system reprogramming privileges based on facial identification. In an example, even if a correct password is entered on the keypad 500, only known persons (e.g., person X or Y) may be allowed to gain access or change access levels or policies, add users, etc.

[0149] refer to Fig.12 , a diagram illustrating an example access control panel using facial recognition is shown. An alternative example access control panel 550 is shown that can utilize the device 100 for facial recognition. The facial recognition can be performed using computer vision operations 254 in conjunction with the depth analysis 252.

[0150] refer to Fig.13, a method (or process) 600 is shown. The method 600 can acquire a structured light pattern using a rolling shutter sensor. The method 600 generally includes a step (or state) 602, a step (or state) 604, a step (or state) 606, a step (or state) 608, a decision step (or state) 610, a step (or state) (612), a decision step (or state) 614, a step (or state) 616, a decision step (or state) 618, and a step (or state) 620.

[0151] Step 602 may begin method 600. In step 604, structured light projector 104 may be inactive (e.g., SL control circuit 112 may not present signal SL TRIG). Next, in step 606, processor 102 may receive an image captured using rolling shutter sensor 130 (e.g., signal VIDEO). In step 608, processor 102 may perform computer vision operation 254 on the captured image. Next, method 600 may move to decision step 610.

[0152] In decision step 610, processor 102 may determine whether object 304 has been detected. In an example, CNN module 190b may determine whether a face is present in the video frame. If an object of interest has not been detected, method 600 may return to step 606. If object of interest 304 has been detected, method 600 may move to step 612. In step 612, processor 102 may determine a start line position 310 and an end line position 312 of object of interest 304 in video frame 300. Start line 310 and end line 312 may correspond to rows of pixels output by rolling shutter sensor 130. Next, method 600 may move to decision step 614.

[0153] In decision step 614, processor 102 may determine whether rolling shutter sensor 130 has reached start line position 310. For example, image signal processor 108 may present one or more signals (e.g., end of frame signal EOF) indicating the line position of rolling shutter sensor 130 in response to signal VIDEO. Timing information about the exposure rate of the lines of rolling shutter sensor 130 may be used to determine the amount of time to reach start line position 310 based on end of frame signal EOF. For example, end of frame signal EOF may be an interrupt signal generated by image signal processor 108. Processor 102 may implement a timer based on a system clock (e.g., implemented by processor 102). The timer may be configured to provide an on time for structured light projector 104. If rolling shutter sensor 130 is not at start line position 310, method 600 may return to decision step 614. If rolling shutter sensor 130 is at start line position 310, method 600 may move to step 616. In step 616 , the SL control circuit 112 may generate a control signal SL TRIG to activate the structured light projector 104 . Next, the method 600 may move to a decision step 618 .

[0154] In decision step 618, the processor 102 may determine whether the rolling shutter sensor 130 has reached the end line position 312. For example, the image signal processor 108 may present one or more signals indicating the line position of the rolling shutter sensor 130, and / or the processor 102 may be configured to perform a calculation based on the exposure time for each row of the rolling shutter sensor (e.g., determine the amount of time from the start line position 310 to the rolling shutter sensor reaching the end line position 312). The processor 102 may implement a timer based on a system clock (e.g., implemented by the processor 102). The timer may be configured to provide a turn-off time for the structured light projector 104. If the rolling shutter sensor 130 is not at the end line position 312, the method 600 may return to decision step 618. If the rolling shutter sensor 130 is at the end line position 312, the method 600 may move to step 620. In step 620, the SL control circuit 112 may generate a control signal SLTRIG to deactivate the structured light projector 104. In some embodiments, the processor 102 may activate the structured light projector 104 earlier than the start line position 310 and / or deactivate the structured light projector 104 later than the end line position 312 to ensure that the region of interest 362 has sufficient integration time. Next, the method 600 may return to step 604.

[0155] refer to Fig.14, a method (or process) 650 is shown. When the structured light pattern is not active, the method 650 can perform computer vision to detect objects to determine the start and end lines of the region of interest in the video frame. The method 650 generally includes step (or state) 652, step (or state) 654, decision step (or state) 656, step (or state) 658, step (or state) 660, step (or state) 662, step (or state) 664 and step (or state) 666.

[0156] Step 652 may begin method 650. In step 654, processor 102 may receive pixel data captured by rolling shutter sensor 130. Next, method 650 may move to decision step 656.

[0157] In decision step 656, processor 102 may determine whether a structured light pattern is captured in the pixel data. For example, processor 102 may determine whether a structured light pattern SLP is captured based on the activation state of structured light projector 104 and / or the state of signal SL TRIG. If the structured light pattern SLP is captured, method 650 may move to step 666. If the structured light pattern SLP is not captured, method 650 may move to step 658.

[0158] In step 658, the processor 102 may analyze the IR image data 202 and / or the RGB image data 204. Next, in step 660, the CNN module 190b may perform computer vision operations 254 and / or facial recognition to detect the object of interest 304. In step 662, the processor 102 may determine the location of the object of interest 304 in the image data. Next, the method 650 may move to step 664.

[0159] In step 664, the processor 102 may determine the start line position 310 and the end line position 312 of the object of interest 304. The processor 102 may use the start line position 310 and the end line position 312 to determine the region of interest 362 when the structured light pattern SLP is active. Next, the method 650 may move to step 666. Step 666 may end the method 650.

[0160] refer to Fig.15 , a method (or process) 700 is shown. When the structured light pattern is in an active state, the method 700 can perform a depth analysis. The method 700 generally includes a step (or state) 702, a step (or state) 704, a decision step (or state) 706, a step (or state) 708, a step (or state) 710, a step (or state) 712, and a step (or state) 714.

[0161] Step 702 may begin method 700. In step 704, processor 102 may receive pixel data captured by rolling shutter sensor 130. Next, method 700 may move to decision step 706.

[0162] In decision step 706, processor 102 may determine whether structured light pattern SLP is captured in the pixel data. For example, processor 102 may determine whether structured light pattern SLP is captured based on the activation state of structured light projector 104 and / or the state of signal SL TRIG. If structured light pattern SLP is not captured, method 700 may move to step 714. If structured light pattern SLP is captured, method 700 may move to step 708.

[0163] In step 708, the processor 102 may analyze the IR image data 202 and / or the RGB image data 204. In the case where the structured light pattern SLP is activated, the processor 102 may receive the IR image data 202 and / or the RGB data 204 and the structured light pattern SLP. Typically, when the structured light pattern SLP is present, the RGB data 204 may be ignored. Next, in step 710, the CNN module 190b may perform a depth analysis 252 in response to the structured light pattern SLP and the IR image data 202. In step 712, the processor 102 may generate a result 256 of the existence determination. Next, the method 700 may move to step 714. Step 714 may end the method 700.

[0164] refer to Fig.16 , a method (or process) 750 is shown. Method 750 can implement a dynamic actuator mechanism for a rolling shutter sensor. Method 750 generally includes step (or state) 752, step (or state) 754, decision step (or state) 756, step (or state) 758, step (or state) 760, step (or state) 762, step (or state) 764, step (or state) 766, decision step (or state) 768, decision step (or state) 770, step (or state) 772, and step (or state) 774.

[0165] Step 752 may start method 750. In step 754, processor 102 may perform computer vision operation 254 on RGB pixel data 204 and / or IR image pixel data 202. Next, method 750 may move to decision step 756.

[0166] In decision step 756, processor 102 may determine whether a face (e.g., object of interest 304) has been detected. If a face has not been detected, method 750 may return to step 754. If a face has been detected, method 750 may move to step 758. In step 758, processor 102 may receive an interrupt signal (e.g., end-of-frame signal EOF) from image signal processor 108. Next, method 750 may move to step 760.

[0167] In step 760, the processor 102 may calculate the exposure time 406 to determine the start line time 310 corresponding to the detected face 304. Next, in step 762, the processor 102 may generate a signal SL TRIG to turn on the structured light projector 104 at the first line 310 where the face 304 is detected. In step 764, the processor 102 may keep the structured light pattern SLP activated to capture the entire facial region of interest 362. Next, in step 766, the processor 102 may turn off the structured light projector 104 (e.g., when the end line 312 has been reached). Next, the method 750 may move to decision step 768.

[0168] In decision step 768, the processor 102 may determine whether the face 304 is still detected. For example, the computer vision operation 254 may be used to analyze the RGB image data 204 and / or the IR image data 202 to detect the location of the face 304. If the face 304 has been detected, the method 768 may move to decision step 770. In decision step 770, the processor 102 may determine whether the location of the face 304 has moved (e.g., whether the face is located in a different location in the captured video frame). If the location of the detected face 304 has not moved, the method 750 may return to step 762. If the location of the detected face 304 has moved, the method 750 may move to step 772. In step 772, the processor 102 may recalculate the starting line location 310. Next, the method 750 may return to step 762.

[0169] In decision step 768, if face 304 is no longer detected, method 750 may move to step 774. Step 774 may end method 750.

[0170] Depend on Figure 1-16The functions performed by the diagrams may be implemented using one or more of a conventional general purpose processor, a digital computer, a microprocessor, a microcontroller, a RISC (Reduced Instruction Set Computer) processor, a CISC (Complex Instruction Set Computer) processor, a SIMD (Single Instruction Multiple Data) processor, a signal processor, a central processing unit (CPU), an arithmetic logic unit (ALU), a video digital signal processor (VDSP), and / or similar computing machines programmed according to the teachings of the specification, as will be apparent to those skilled in the relevant art(s). A skilled programmer can readily prepare appropriate software, firmware, coding, routines, instructions, opcodes, microcodes, and / or program modules based on the teachings of the present disclosure, as will be apparent to those skilled in the relevant art(s). The software is typically executed from one or more media by one or more of the processors of the machine implementation.

[0171] The present invention may also be implemented by preparing an ASIC (application specific integrated circuit), a platform ASIC, an FPGA (field programmable gate array), a PLD (programmable logic device), a CPLD (complex programmable logic device), a sea-of-gate, an RFIC (radio frequency integrated circuit), an ASSP (application specific standard product), one or more monolithic integrated circuits, one or more chips or dies arranged as a flip-chip module and / or a multi-chip module, or by interconnecting an appropriate network of conventional component circuits, as described herein, modifications of which will be apparent to those skilled in the art.

[0172] Thus, the present invention may also include a computer product, which may be one or more storage media and / or one or more transmission media, which may include instructions that may be used to program a machine to perform one or more processes or methods according to the present invention. The execution of the instructions contained in the computer product by the machine and the operation of the surrounding circuitry may convert input data into one or more files on the storage medium and / or one or more output signals representing a physical object or substance, such as audio and / or visual depictions. The storage medium may include, but is not limited to, any type of disk, including floppy disks, hard drives, magnetic disks, optical disks, CD-ROMs, DVDs, and magneto-optical disks, as well as circuits such as ROM (read-only memory), RAM (random access memory), EPROM (erasable programmable ROM), EEPROM (electrically erasable programmable ROM), UVPROM (ultraviolet erasable programmable ROM), flash memory, magnetic cards, optical cards, and / or any type of medium suitable for storing electronic instructions.

[0173] The element of the present invention can form a part or all of one or more equipment, units, components, systems, machines and / or devices. Equipment can include but is not limited to servers, workstations, storage array controllers, storage systems, personal computers, laptop computers, notebook computers, palmtop computers, cloud servers, personal digital assistants, portable electronic devices, battery-powered equipment, set-top boxes, encoders, decoders, code converters, compressors, decompressors, preprocessors, postprocessors, transmitters, receivers, transceivers, cryptographic circuits, cellular phones, digital cameras, positioning and / or navigation systems, medical equipment, heads-up displays, wireless devices, audio recordings, audio storage and / or audio playback equipment, video recordings, video storage and / or video playback equipment, gaming platforms, peripherals and / or multi-chip modules. (Multiple) technicians in the relevant field will understand that the element of the present invention can be implemented in other types of equipment to meet the standards of specific applications.

[0174] When used in conjunction with "is" and a verb herein, the terms "may" and "generally" are intended to convey the intention that the description is exemplary and is considered broad enough to cover both the specific examples presented in the present disclosure and alternative examples that can be derived based on the present disclosure. The terms "may" and "generally" as used herein should not be interpreted as necessarily implying the desirability or possibility of omitting the corresponding elements.

[0175] While the invention has been particularly shown and described with reference to embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the scope of the invention.

Claims

1. An apparatus for controlling activation of a structured light pattern, comprising: a rolling shutter image sensor configured to capture an image; a structured light projector configured to generate a structured light pattern; as well as a processor configured to: (i) receive the image from the rolling shutter image sensor; (ii) perform computer vision operations on the image to detect an object in the image; (iii) determine a start line location and an end line location corresponding to the detected object; and (iv) present a control signal to the structured light projector, wherein the control signal is configured to: (a) activate the structured light pattern when the rolling shutter image sensor is at the start line location; and (b) when the rolling shutter image sensor is located at the end line position, deactivating the structured light pattern.

2. The device according to claim 1, wherein: The control signal is configured to enable the rolling shutter image sensor to capture the structured light pattern at a specific area corresponding to the detected object.

3. The device according to claim 1, wherein: The processor is also configured to generate a depth image using the structured light pattern.

4. The device according to claim 3, wherein: The depth image is analyzed by the processor to perform at least one of 3D sensing, 3D modeling, 3D facial recognition, and presence determination.

5. The device according to claim 1, wherein: The detected object is a human face.

6. The device according to claim 1, wherein: The region of interest is located between the start line position and the end line position of the image.

7. The device according to claim 6, wherein: In response to detecting that the position of the object changes over time, the region of interest is dynamically adjusted.

8. The device according to claim 1, wherein: The rolling shutter image sensor includes an RGB-IR sensor.

9. The device according to claim 1, wherein: (i) the processor also includes an image signal processing circuit; (ii) the image signal processing circuit is configured to: (a) generate an interrupt signal in response to the image received from the rolling shutter image sensor; (iii) in response to the interrupt signal, the processor is also configured to: (a) calculate the exposure time to determine the start line position; and (b) implement a timer based on a system clock; and (iv) the timer is configured to provide a start time for the structured light projector after the object is detected.

10. The device according to claim 9, wherein: The interrupt signal includes an indication of the start line position.

11. The device according to claim 9, wherein: The interrupt signal includes an indication of the end line position.

12. The device according to claim 9, wherein: The interrupt signal includes data corresponding to a fixed frame rate to enable the processor to calculate the time of the exposure to determine the starting line position.

13. The device according to claim 9, wherein: The interrupt signal and the timer are implemented to control the timing of activation of the structured light pattern to match the exposure timing of the image.

14. The device according to claim 1, wherein: The device is implemented as part of a low power camera.

15. The apparatus of claim 1, further comprising a second rolling shutter sensor, wherein: (a) the rolling shutter image sensor comprises an RGB rolling shutter sensor; and (b) the second rolling shutter sensor comprises an IR rolling shutter sensor configured to capture an IR image.

16. The apparatus of claim 1, further comprising an image signal processing circuit configured to process the image captured by the rolling shutter image sensor, wherein: The image signal processing circuit separates the image into color image data and IR image data.

17. The device according to claim 16, wherein: The processor is further configured to: (i) analyze the IR image data when the structured light pattern is activated to obtain 3D information for a field of view of the rolling shutter image sensor; and (ii) analyzing the IR image data and the color image data when the structured light pattern is deactivated to perform the computer vision operation to determine the start line location and the end line location of the detected object.

18. The device according to claim 1, wherein: The processor is further configured to crop a portion of the image above the start line location and below the end line location when the structured light pattern is activated.

19. A method for acquiring a structured light pattern using a rolling shutter image sensor, comprising the following steps: (A) capturing an image using the rolling shutter image sensor; (B) receiving the image at a processor; (C) performing computer vision operations on the image to detect objects in the image; (D) determining a start line position and an end line position corresponding to the detected object; (E) presenting a control signal to the structured light projector; as well as (F) generating the structured light pattern using the structured light projector, wherein the control signal is configured to: (a) activate the structured light pattern when the rolling shutter image sensor is located at the start line position; and (b) when the rolling shutter image sensor is located at the end line position, deactivating the structured light pattern.

Images