Adaptive face depth map generation

Abstract

An apparatus includes an interface, a light projector, and a processor. The interface can be configured to receive pixel data. The light projector can be configured to generate a structured light pattern. The processor can be configured to process the pixel data arranged as a video frame including the structured light pattern, perform a computer vision operation to detect a size of a face region of the video frame, determine a scaling ratio in response to the size of the face region, extract the structured light pattern from the video frame, generate a reduced structured light image, and generate a depth map in response to the reduced structured light image and a reduced reference image. A reduction operation can be performed in response to the scaling ratio to generate the reduced structured light image. The scaling ratio can enable generation of the reduced structured light image with sufficient depth pixels.

Description

TECHNICAL FIELD

[0001] The present invention relates generally to computer vision, and more particularly, the present invention relates to a method and / or apparatus for implementing adaptive face depth map generation. BACKGROUND

[0002] Machine vision, optical technology, and artificial intelligence have been rapidly developed. Due to the advancement of robotics and deep learning technology, three-dimensional (3D) reconstruction has become an important branch of machine vision. Traditional 3D reconstruction technology has problems in real-time applications. The speed of 3D reconstruction is not fast enough, and the accuracy of 3D reconstruction is not accurate enough.

[0003] There are two main methods of 3D reconstruction based on machine vision: active vision method and passive vision method. The active vision method includes time-of-flight method, structured light method, holographic interference method, Moire method, etc. The passive vision method includes binocular stereo vision, motion vision, defocus method, shading method, focusing method, etc.

[0004] One method of active 3D reconstruction is performed by generating a depth map using a monocular speckle structured light system. Due to the insufficient speed and accuracy of 3D reconstruction, applications are generally limited to scenarios where accuracy is not required, such as three-dimensional face recognition, and face liveness detection, etc.

[0005] The speed of depth map generation using structured light is closely related to the resolution of the depth map. Depth map calculation usually uses a block matching method. The calculation time of the depth map using block matching is linearly proportional to the input size of the structured light image. Therefore, compared to depth map calculation on a smaller face area (i.e., 100x100 pixels), depth map calculation on a larger face area (i.e., 500x500 pixels) will take 25 times longer.

[0006] A larger face area is usually generated when capturing a face close to the camera sensor, while a smaller face area is usually generated when capturing a face far from the camera sensor. For example, if the face is 0.3 meters away from a 1920x1080 camera, the face area can include up to 0.5 megapixels. Performing calculations to generate a depth map of a 0.5 megapixel face area can take about hundreds of milliseconds. Hundreds of milliseconds of calculation time is not acceptable in a real product. On the other hand, performing calculations to generate a depth map of a tiny face area only takes a few milliseconds. The inconsistent performance of face areas of different sizes is a problem faced in product development and user experience. The calculation speed varies depending on the face area. In addition, various resolutions of face depth maps become a problem for liveness algorithm training.

[0007] Liveness detection algorithms use a series of required amounts of depth information to implement to distinguish between a real human face and a fake human face. Large face regions that provide a large amount of depth information can exceed the input requirements of liveness detection, resulting in unnecessary computations. Unnecessary computations increase the time to create a depth map and not only increase the workload of depth map computation, but also increase the overall workload of liveness detection.

[0008] It would be desirable to implement adaptive face depth map generation. SUMMARY

[0009] An apparatus includes an interface, a structured light projector, and a processor. The interface can be configured to receive pixel data. The structured light projector can be configured to generate a structured light pattern. The processor can be configured to process the pixel data arranged as a video frame including the structured light pattern, perform a computer vision operation on the video frame to detect a size of a face region of the video frame, determine a scaling ratio in response to the size of the face region, extract the structured light pattern from the video frame, generate a reduced structured light image, and generate a depth map in response to the reduced structured light image and a reduced reference image. A reduction operation can be performed in response to the scaling ratio and the structured light pattern to generate the reduced structured light image. The scaling ratio can be calculated to enable generation of the reduced structured light image with a sufficient number of depth pixels. BRIEF DESCRIPTION OF DRAWINGS

[0010] Embodiments of the application will become apparent from the following detailed description and accompanying drawings.

[0011] Figure 1 FIG. 1 is a diagram illustrating an example of an edge device that can utilize a processor configured to implement adaptive face depth map generation according to example embodiments of the application.

[0012] Figure 2 FIG. 2 is a diagram illustrating an example camera implementing example embodiments of the application.

[0013] Figure 3 FIG. 3 is a diagram illustrating a block diagram of a camera system.

[0014] Figure 4 FIG. 4 is a diagram illustrating processing circuitry of a camera system implementing a convolutional neural network configured to perform 3D reconstruction.

[0015] Figure 5 FIG. 5 is a diagram illustrating an example video frame including a large face region and a small face region.

[0016] Figure 6This is a block diagram illustrating the operation of the processor used for adaptive facial depth map generation.

[0017] Figure 7 This is a diagram showing the extraction of IR video frames for downsampling.

[0018] Figure 8 This is a diagram showing a reduced structured light pattern.

[0019] Figure 9 This is a diagram showing the reduction of IR video frames.

[0020] Figure 10 This is a flowchart illustrating a method for generating adaptive facial depth maps.

[0021] Figure 11 This is a flowchart illustrating a method for determining the scaling ratio of each video frame.

[0022] Figure 12 This is a flowchart illustrating a method for performing post-processing on a depth map with consistent target dimensions.

[0023] Figure 13 This is a flowchart illustrating a method for pre-calculating a reduced reference image. Detailed Implementation

[0024] Embodiments of the present invention include providing adaptive facial depth map generation, which can (i) reduce the amount of computation required to generate the depth map, (ii) enable real-time liveness detection, (iii) provide consistent depth map computation time, (iv) provide consistent depth maps for training liveness detection, (v) reduce the structured light pattern image while ensuring a sufficient number of depth pixels, (vi) provide sufficient depth pixels based on an expected range of the number of depth pixels used for liveness detection, (vii) dynamically create depth maps at various resolutions, (viii) calculate a scaling ratio in response to the size of the facial region and the expected number of depth pixels, (ix) perform a reduction operation that increases computational cost less than the computational cost reduced for generating the depth map from the reduced image, and / or (vi) be implemented as one or more integrated circuits.

[0025] Embodiments of the present invention can be configured to adaptively generate depth maps. In an example, an adaptive depth map can be generated for facial liveness detection. The resolution of the depth map can depend on the resolution of the real-time structured light pattern image. The real-time structured light pattern image and the reference structured light image can be downsized. By downscaling the real-time structured light pattern image and the reference structured light image, a depth map of any resolution can be dynamically created. An adaptive depth map of any resolution can be dynamically created in response to captured faces of different sizes.

[0026] Adaptive depth map generation can reduce depth computation time compared to generating depth maps using full-resolution structured light images (e.g., real-time and reference light images). Adaptive depth map generation can reduce the complexity of performing liveness detection techniques with fixed-depth facial resolution (e.g., consistent facial depth density). Typically, liveness detection techniques can operate using a range of depth information considered sufficient to distinguish between real and fake faces. The adaptive depth maps generated by embodiments of the present invention can consistently create facial depth with a sufficient number of depth pixels for liveness detection.

[0027] A sufficient number of depth pixels in an adaptive depth map falls within the range of depth pixels required for liveness detection techniques. This allows for a sufficient number of depth pixels to avoid unnecessary computations when providing depth maps based on large facial regions. Reducing the structured light pattern image can provide a sufficient number of depth pixels, which reduces the amount of computation performed on both the depth map and liveness detection. Consistently creating facial depth maps with a sufficient number of depth pixels prevents wasted time computing excessive depth information (e.g., avoids unnecessary computations).

[0028] The amount of reduction performed on the real-time structured light pattern image and / or reference image can be selected to ensure successful block matching. For example, reduction can be intelligently selected to allow for a decrease in accuracy that does not lead to block matching failure. Block matching failure may occur in response to a large reduction factor that results in the removal of specific features of the structured light pattern. Embodiments of the invention allow control over the scaling factor used for the reduction operation. The scaling factor can be controlled within an acceptable range (e.g., [1, 16]) to ensure that block matching of the real-time structured light pattern is not affected.

[0029] The decrease in accuracy can be proportional to the scaling factor on the image width used for the reduction operation. The granularity of facial parallax can be related to the width resolution. The closer an object (e.g., a face) is to the image sensor, the more parallax information is available. Large facial regions resulting from the proximity of the face to the image sensor may have larger scaling and larger parallax values. Larger parallax values ​​can compensate for the accuracy loss that may occur during the reduction operation. Users can adjust the scaling factor used to provide the reduction ratio. Embodiments of the invention can provide tolerance parameters. Tolerance parameters allow users to adjust the balance between computation speed and depth map size.

[0030] Embodiments of the present invention can provide consistent performance and / or depth map results. Consistent depth maps can be beneficial for training liveness detection techniques. Consistent depth maps can be used with liveness detection techniques to provide accurate detection. For example, liveness detection techniques can focus on optimizing detection for a specific facial depth resolution, rather than covering and training for all cases with different (e.g., multiple) facial depth resolutions.

[0031] Embodiments of the present invention can be configured to determine facial regions in an image. Face detection can be performed to determine the offset and size of the detected face. A predetermined number of depth pixels for a specific liveness detection technique can be taken. Tolerance parameters can be adjusted by the end user (e.g., within a limit on the predetermined number of depth pixels) to balance between detection accuracy and computational speed. A scaling ratio can be calculated based on the size of the facial region, the predetermined depth pixels, and the tolerance parameters. Generally, the larger the facial region, the smaller the result of the scaling ratio calculation. A reduction operation can be performed based on the scaling ratio. The scaling ratio can be used to reduce a high-resolution real-time structured light image and a high-resolution reference structured light image. After performing the reduction operation, the reduced real-time structured light image and the reduced reference structured light image can be used to generate a depth map. The additional resources required to perform the reduction operation may be less than the amount of resources saved by reducing the complexity of depth map computation in response to generating a depth map using a reduced structured light image. Using a reduced structured light image can reduce facial depth computation time and provide consistent facial depth computation compared to using a full-resolution structured light image. Consistent facial depth map input can benefit post-processing operations.

[0032] refer to Figure 1 , Figure 1 A diagram illustrating an example of an edge device, according to an exemplary embodiment of the invention, that can utilize a processor configured to implement adaptive facial depth map generation. A top view of region 50 is shown. In the illustrated example, region 50 may be an outdoor location. Streets, vehicles, and buildings are shown.

[0033] Devices 100a-100n are shown at different locations within region 50. Each of devices 100a-100n can independently implement an edge device. Edge devices 100a-100n may include smart IP cameras (e.g., camera systems). Edge devices 100a-100n may include low-power technologies (e.g., microprocessors running on sensors, cameras, or other battery-powered devices) in embedded platforms designed for deployment at the network edge, where power consumption is a critical issue. In the example, edge devices 100a-100n may include various traffic cameras and Intelligent Transportation System (ITS) solutions.

[0034] Edge devices 100a-100n can be implemented for a variety of applications. In the illustrated example, edge devices 100a-100n may include an Automatic License Plate Recognition (ANPR) camera 100a, a traffic camera 100b, a vehicle camera 100c, an access control camera 100d, an ATM camera 100e, a bullet camera 100f, a dome camera 100n, etc. In the example, edge devices 100a-100n can be implemented as a traffic camera and Intelligent Transportation System (ITS) solution designed to enhance road safety by leveraging a combination of people and vehicle detection, vehicle brand / model recognition, and Automatic License Plate Recognition (ANPR) capabilities.

[0035] In the illustrated example, region 50 can be an outdoor location. In some embodiments, edge devices 100a-100n can be implemented in various indoor locations. In the example, edge devices 100a-100n can incorporate convolutional neural networks for use in security (surveillance) applications and / or access control applications. In the example, edge devices 100a-100n implemented as security cameras and access control applications can include battery-powered cameras, doorbell cameras, outdoor cameras, indoor cameras, etc. According to embodiments of the invention, security camera and access control applications can achieve performance advantages from the application of convolutional neural networks. In the example, edge devices utilizing convolutional neural networks according to embodiments of the invention can acquire large amounts of image data and perform on-device inference to obtain useful information (e.g., multiple temporal instances of images performed by each network), thereby reducing bandwidth and / or power consumption. The design, type, and / or application performed by edge devices 100a-100n can vary depending on the design criteria of a particular implementation.

[0036] refer to Figure 2 The diagram illustrates an example edge device camera that implements an exemplary embodiment of the present invention. Camera systems 100a-100n are shown. Each camera device 100a-100n can have a different style and / or use case. For example, camera 100a can be an action camera, camera 100b can be a ceiling-mounted security camera, camera 100n can be a webcam, etc. Other types of cameras can be implemented (e.g., home security cameras, battery-powered cameras, doorbell cameras, stereo cameras, etc.). The design / style of cameras 100a-100n can vary depending on the design criteria of a particular implementation.

[0037] Each of the camera systems 100a-100n may include block (or circuit) 102, block (or circuit) 104, and / or block (or circuit) 106. Circuit 102 may implement a processor. Circuit 104 may implement a capture device. Circuit 106 may implement a structured light projector. Camera systems 100a-100n may include other components (not shown). They can be used with... Figure 3 The details of the components of the cameras 100a-100n are described in relation to each other.

[0038] Processor 102 can be configured to implement an artificial neural network (ANN). In an example, the ANN may include a convolutional neural network (CNN). Processor 102 can be configured to implement a video encoder. Processor 102 can be configured to process pixel data arranged into video frames. Capture device 104 can be configured to capture pixel data that can be used by processor 102 to generate video frames. Structured light projector 106 can be configured to generate a structured light pattern (e.g., a speckle pattern). The structured light pattern can be projected onto a background (e.g., the environment). Capture device 104 can capture pixel data including a background image (e.g., the environment) with a speckle pattern.

[0039] Cameras 100a-100n can be edge devices. A processor 102 implemented by each of the cameras 100a-100n enables the cameras 100a-100n to perform various functions internally (e.g., at the local level). For example, processor 102 can be configured to perform on-device object / event detection (e.g., computer vision operations), 3D reconstruction, liveness detection, depth map generation, video encoding, and / or video transcoding. For example, processor 102 can even perform advanced processes such as computer vision and 3D reconstruction without uploading video data to a cloud service to offload computationally intensive functions (e.g., computer vision, video encoding, video transcoding, etc.).

[0040] In some embodiments, multiple camera systems may be implemented (e.g., camera systems 100a-100n may operate independently of each other). For example, each of cameras 100a-100n may individually analyze captured pixel data and perform event / object detection locally. In some embodiments, cameras 100a-100n may be configured as a camera network (e.g., security cameras that send video data to a central source such as network-attached storage and / or cloud services). The location and / or configuration of cameras 100a-100n may vary depending on the design criteria of the specific implementation.

[0041] The capture device 104 of each of the camera systems 100a-100n may include a single lens (e.g., a monocular camera). The processor 102 may be configured to accelerate the preprocessing of speckle structured light for monocular 3D reconstruction. Monocular 3D reconstruction can be performed to generate depth maps and / or parallax images without using a stereo camera.

[0042] refer to Figure 3 The diagram illustrates a block diagram of a camera system 100 implemented as an example. The camera system 100 can be a combination of... Figure 2 Representative examples of cameras 100a-100n are shown. Camera system 100 may include processor / SoC 102, capture device 104, and structured light projector 106.

[0043] The camera system 100 may further include blocks (or circuits) 150, 152, 154, 156, 158, 160, 162, 164, and / or 166. Circuit 150 may implement a memory. Circuit 152 may implement a battery. Circuit 154 may implement a communication device. Circuit 156 may implement a wireless interface. Circuit 158 ​​may implement a general-purpose processor. Block 160 may implement an optical lens. Block 162 may implement a structured light patterning lens. Circuit 164 may implement one or more sensors. Circuit 166 may implement a human-machine interface (HID) device. In some embodiments, the camera system 100 may include a processor / SoC 102, a capture device 104, an IR structured light projector 106, a memory 150, a lens 160, an IR structured light projector 106, a structured light pattern lens 162, a sensor 164, a battery 152, a communication module 154, a wireless interface 156, and a processor 158. In another example, the camera system 100 may include the processor / SoC 102, the capture device 104, the structured light projector 106, the processor 158, the lens 160, the structured light pattern lens 162, and the sensor 164 as a single device, and the memory 150, the battery 152, the communication module 154, and the wireless interface 156 may be components of separate devices. The camera system 100 may include other components (not shown). The number, type, and / or arrangement of the components of the camera system 100 may vary depending on the design criteria of a particular implementation.

[0044] Processor 102 can be implemented as a video processor. In an example, processor 102 can be configured to receive three-sensor video input using a high-speed SLVS / MIPI-CSI / LVCMOS interface. In some embodiments, processor 102 can be configured to perform depth sensing in addition to generating video frames. In an example, depth sensing can be performed in response to depth information and / or vector light data captured in the video frames.

[0045] Memory 150 can store data. Memory 150 can be implemented in various types of memory, including but not limited to cache, flash memory, memory cards, random access memory (RAM), dynamic RAM (DRAM), etc. The type and / or size of memory 150 can vary depending on the design criteria of a particular implementation. The data stored in memory 150 may correspond to video files, motion information (e.g., readings from sensor 164), video fusion parameters, image stabilization parameters, user input, computer vision models, feature sets, and / or metadata information. In some embodiments, memory 150 can store reference images. Reference images can be used for computer vision operations, 3D reconstruction, etc. In some embodiments, reference images may include reference structured light images.

[0046] The processor / SoC 102 can be configured to execute computer-readable code and / or process information. In various embodiments, the computer-readable code may be stored within the processor / SoC 102 (e.g., microcode, etc.) and / or in memory 150. In an example, the processor / SoC 102 can be configured to execute one or more artificial neural network models (e.g., face recognition CNN, object detection CNN, object classification CNN, 3D reconstruction CNN, liveness detection CNN, etc.) stored in memory 150. In an example, memory 150 may store one or more directed acyclic graphs (DAGs) and one or more sets of weights and biases defining one or more artificial neural network models. The processor / SoC 102 can be configured to receive input from memory 150 and / or present output to memory 150. The processor / SoC 102 can be configured to present and / or receive other signals (not shown). The number and / or type of inputs and / or outputs of the processor / SoC 102 may vary depending on the design criteria of a particular implementation. The processor / SoC 102 can be configured for low-power (e.g., battery) operation.

[0047] Battery 152 can be configured to store and / or power components of camera system 100. The dynamic driver mechanism for the rolling shutter sensor can be configured to conserve power. Reduced power consumption allows camera system 100 to operate for extended periods using battery 152 without recharging. Battery 152 can be rechargeable. Battery 152 can be built-in (e.g., non-replaceable) or replaceable. Battery 152 can have an input for connecting to an external power source (e.g., for charging). In some embodiments, device 100 can be powered by an external power source (e.g., battery 152 can be omitted or implemented as a backup power source). Various battery technologies and / or chemistry can be used to implement battery 152. The type of battery 152 implemented can vary depending on the design criteria of a particular implementation.

[0048] The communication module 154 can be configured to implement one or more communication protocols. For example, the communication module 154 and the wireless interface 156 can be configured to implement one or more of the following: IEEE 102.11, IEEE 102.15, IEEE 102.15.1, IEEE 102.15.2, IEEE 102.15.3, IEEE 102.15.4, IEEE 102.15.5, IEEE 102.20, etc. and / or In some embodiments, the communication module 154 may be a hardwired data port (e.g., a USB port, a mini-USB port, a USB-C connector, an HDMI port, an Ethernet port, a DisplayPort interface, a Lightning port, etc.). In some embodiments, the wireless interface 156 may also implement one or more protocols associated with a cellular communication network (e.g., GSM, CDMA, GPRS, UMTS, CDMA2000, 3GPP LTE, 4G / HSPA / WiMAX, SMS, etc.). In embodiments where the camera system 100 is implemented as a wireless camera, the protocol implemented by the communication module 154 and the wireless interface 156 may be a wireless communication protocol. The type of communication protocol implemented by the communication module 154 may vary depending on the design criteria of the specific implementation.

[0049] Communication module 154 and / or wireless interface 156 can be configured to generate broadcast signals as output from camera system 100. The broadcast signals can send video data, parallax data, and / or control signals to external devices. For example, the broadcast signals can be sent to cloud storage services (e.g., storage services capable of scaling on demand). In some embodiments, communication module 154 may not send data until the processor / SoC 102 has performed video analysis to determine that an object is in the field of view of camera system 100.

[0050] In some embodiments, the communication module 154 can be configured to generate a manual control signal. The manual control signal can be generated in response to a signal received from a user by the communication module 154. The manual control signal can be configured to activate the processor / SoC 102. Regardless of the power state of the camera system 100, the processor / SoC 102 can be activated in response to the manual control signal.

[0051] In some embodiments, the communication module 154 and / or the wireless interface 156 may be configured to receive a feature set. The received feature set can be used to detect events and / or objects. For example, the feature set can be used to perform computer vision operations. The feature set information may include instructions for the processor 102 to determine which types of objects correspond to objects and / or events of interest.

[0052] In some embodiments, the communication module 154 and / or the wireless interface 156 may be configured to receive user input. User input allows the user to adjust operating parameters of various features implemented by the processor 102. In some embodiments, the communication module 154 and / or the wireless interface 156 may be configured to interface with an application (e.g., an app) (e.g., using an application programming interface (API)). For example, the application may be implemented on a smartphone to enable the end user to adjust various settings and / or parameters for various features implemented by the processor 102 (e.g., setting video resolution, selecting frame rate, selecting output format, setting tolerance parameters for 3D reconstruction, etc.).

[0053] Processor 158 can be implemented using general-purpose processor circuitry. Processor 158 is operable to interact with video processing circuitry 102 and memory 150 to perform various processing tasks. Processor 158 can be configured to execute computer-readable instructions. In this example, the computer-readable instructions may be stored in memory 150. In some embodiments, the computer-readable instructions may include controller operations. Typically, input from sensor 164 and / or human-machine interface device 166 is shown to be received by processor 102. In some embodiments, general-purpose processor 158 can be configured to receive and / or analyze data from sensor 164 and / or HID 166 and make decisions in response to the input. In some embodiments, processor 158 can send data to and / or receive data from other components of camera system 100, such as battery 152, communication module 154, and / or wireless interface 156. Which functions of camera system 100 are performed by processor 102 and general-purpose processor 158 may vary depending on the design criteria of the specific implementation.

[0054] Lens 160 may be attached to capture device 104. Capture device 104 may be configured to receive an input signal (e.g., LIN) via lens 160. The signal LIN may be an optical input (e.g., an analog image). Lens 160 may be implemented as an optical lens. Lens 160 may provide zoom and / or focus features. In one example, capture device 104 and / or lens 160 may be implemented as a single lens assembly. In another example, lens 160 may be implemented separately from capture device 104.

[0055] Capture device 104 can be configured to convert input light LIN into computer-readable data. Capture device 104 can capture data received through lens 160 to generate raw pixel data. In some embodiments, capture device 104 can capture data received through lens 160 to generate a bitstream (e.g., generate video frames). For example, capture device 104 can receive focused light from lens 160. Lens 160 can be oriented, tilted, translated, scaled, and / or rotated to provide a target view from camera system 100 (e.g., a view of video frames, a view of panoramic video frames captured using multiple camera systems 100a-100n, a target image and reference image view for stereo vision, etc.). Capture device 104 can generate a signal (e.g., video). The signal VIDEO can be pixel data (e.g., a sequence of pixels that can be used to generate video frames). In some embodiments, the signal VIDEO can be video data (e.g., a sequence of video frames). The signal VIDEO can be presented to one of the inputs of processor 102. In some embodiments, the pixel data generated by the capture device 104 may be uncompressed and / or raw data generated in response to focused light from the lens 160. In some embodiments, the output of the capture device 104 may be a digital video signal.

[0056] In this example, capture device 104 may include block (or circuitry) 180, block (or circuitry) 182, and block (or circuitry) 184. Circuitry 180 may be an image sensor. Circuitry 182 may be a processor and / or logic unit. Circuitry 184 may be memory circuitry (e.g., a frame buffer). Lens 160 (e.g., a camera lens) may be directed to provide a view of the environment surrounding camera system 100. Lens 160 may be designed to capture ambient data (e.g., light input LIN). Lens 160 may be a wide-angle lens and / or a fisheye lens (e.g., a lens capable of capturing a wide field of view). Lens 160 may be configured to capture and / or focus light for capture device 104. Typically, image sensor 180 is located behind lens 160. Based on the light captured from lens 160, capture device 104 may generate bitstream and / or video data (e.g., a signal VIDEO).

[0057] Capture device 104 can be configured to capture video image data (e.g., light collected and focused by lens 160). Capture device 104 can capture data received through lens 160 to generate a video bitstream (e.g., pixel data of a video frame sequence). In various embodiments, lens 160 can be implemented as a fixed-focus lens. Fixed-focus lenses are generally advantageous for smaller size and lower power. In examples, fixed-focus lenses can be used in battery-powered, doorbell, and other low-power camera applications. In some embodiments, lens 160 can be oriented, tilted, panned, zoomed, and / or rotated to capture the environment around camera system 100 (e.g., capture data from the field of view). In examples, professional camera models can be implemented using an active lens system for enhanced functionality, remote control, etc.

[0058] The capture device 104 can convert received light into a digital data stream. In some embodiments, the capture device 104 can perform analog-to-digital conversion. For example, the image sensor 180 can perform photoelectric conversion on the light received by the lens 160. The processor / logic unit 182 can convert the digital data stream into a video data stream (or bitstream), a video file, and / or multiple video frames. In the example, the capture device 104 can present the video data as a digital video signal (e.g., VIDEO). The digital video signal may include video frames (e.g., continuous digital images and / or audio). In some embodiments, the capture device 104 may include a microphone for capturing audio. In some embodiments, the microphone may be implemented as a separate component (e.g., one of the sensors 164).

[0059] Video data captured by capture device 104 can be represented as a signal / bitstream / data VIDEO (e.g., a digital video signal). Capture device 104 can present the signal VIDEO to processor / SoC 102. The signal VIDEO can represent video frames / video data. The signal VIDEO can be a video stream captured by capture device 104. In some embodiments, the signal VIDEO may include pixel data operable by processor 102 (e.g., a video processing pipeline, image signal processor (ISP), etc.). Processor 102 can generate video frames in response to the pixel data in the signal VIDEO.

[0060] The signal VIDEO may include pixel data arranged as video frames. The signal VIDEO may be an image including a background (e.g., captured objects and / or environment) and a speckle pattern generated by the structured light projector 106. The signal VIDEO may include a single-channel source image. A single-channel source image may be generated in response to capturing pixel data using the monocular lens 160.

[0061] Image sensor 180 can receive input light LIN from lens 160 and convert the light LIN into digital data (e.g., a bitstream). For example, image sensor 180 can perform photoelectric conversion on light from lens 160. In some embodiments, image sensor 180 may have additional margins that are not used as part of the image output. In some embodiments, image sensor 180 may not have additional margins. In various embodiments, image sensor 180 can be configured to generate RGB-IR video signals. In a field of view illuminated only by infrared light, image sensor 180 can generate monochrome (B / W) video signals. In a field of view illuminated by both IR and visible light, image sensor 180 can be configured to generate color information in addition to generating monochrome video signals. In various embodiments, image sensor 180 can be configured to generate video signals in response to visible light and / or infrared (IR) light.

[0062] In some embodiments, the camera sensor 180 may include a rolling shutter sensor or a global shutter sensor. In an example, the rolling shutter sensor 180 may be implemented as an RGB-IR sensor. In some embodiments, the capture device 104 may include a rolling shutter IR sensor and an RGB sensor (e.g., implemented as separate components). In an example, the rolling shutter sensor 180 may be implemented as an RGB-IR rolling shutter complementary metal-oxide-semiconductor (CMOS) image sensor. In one example, the rolling shutter sensor 180 may be configured to assert a signal indicating the exposure time of the first row. In one example, the rolling shutter sensor 180 may apply a mask to a monochrome sensor. In an example, the mask may include multiple cells containing a red pixel, a green pixel, a blue pixel, and an IR pixel. The IR pixel may contain red, green, and blue filter materials that efficiently absorb all light in the visible spectrum while allowing longer infrared wavelengths to pass through with minimal loss. In the case of a rolling shutter, as each row (or line) of the sensor begins exposure, all pixels in that row (or line) may begin exposure simultaneously.

[0063] Processor / logic unit 182 can convert the bitstream into human-readable content (e.g., video data that an average person can understand regardless of image quality, such as video frames and / or pixel data that can be converted into video frames by processor 102). For example, processor / logic unit 182 can receive raw (e.g., raw) data from image sensor 180 and generate (e.g., encode) video data (e.g., bitstream) based on the raw data. Capture device 104 may have memory 184 to store raw data and / or processed bitstreams. For example, capture device 104 may implement frame memory and / or buffer 184 to store (e.g., provide temporary storage and / or cache) one or more video frames (e.g., digital video signals). In some embodiments, processor / logic unit 182 can perform analysis and / or correction on the video frames stored in memory / buffer 184 of capture device 104. Processor / logic unit 182 can provide status information about the captured video frames.

[0064] The structured light projector 106 may include a block (or circuit) 186. Circuit 186 may implement a structured light source. The structured light source 186 may be configured to generate a signal (e.g., a speckle pattern). The signal SLP may be a structured light pattern (e.g., a speckle pattern). The signal SLP may be projected onto the environment near the camera system 100. The structured light pattern SLP may be captured by a capture device 104 as part of a light input LIN.

[0065] The structured light patterning lens 162 can be a lens for the structured light projector 106. The structured light patterning lens 162 can be configured to allow the structured light SLP generated by the structured light source 186 of the structured light projector 106 to be emitted, while protecting the structured light source 186. The structured light patterning lens 162 can be configured to decompose the laser pattern generated by the structured light source 186 into a pattern array (e.g., a dense dot pattern array for speckle patterns).

[0066] In the example, the structured light source 186 can be implemented as an array of vertical cavity surface-emitting lasers (VCSELs) and lenses. However, other types of structured light sources can be implemented to meet the design criteria of specific applications. In the example, the VCSEL array is typically configured to generate laser patterns (e.g., signal SLP). The lenses are typically configured to decompose the laser pattern into an array of dense dot patterns. In the example, the structured light source 186 can be implemented as a near-infrared (NIR) light source. In various embodiments, the light source of the structured light source 186 can be configured to emit light with a wavelength of approximately 940 nanometers (nm), which is invisible to the human eye. However, other wavelengths can be utilized. In the example, wavelengths in the range of approximately 800 nm to 1000 nm can be utilized.

[0067] Sensor 164 can implement multiple sensors, including but not limited to motion sensors, ambient light sensors, proximity sensors (e.g., ultrasonic, radar, lidar, etc.), audio sensors (e.g., microphones), etc. In embodiments implementing a motion sensor, sensor 164 can be configured to detect motion anywhere (or some location outside the field of view) monitored by camera system 100. In various embodiments, motion detection can be used as a threshold for activating capture device 104. Sensor 164 can be implemented as an internal component of camera system 100 and / or an external component of camera system 100. In one example, sensor 164 can be implemented as a passive infrared (PIR) sensor. In another example, sensor 164 can be implemented as a smart motion sensor. In yet another example, sensor 164 can be implemented as a microphone. In embodiments implementing a smart motion sensor, sensor 164 may include a low-resolution image sensor configured to detect motion and / or people.

[0068] In various embodiments, sensor 164 may generate signals (e.g., SENS). The SENS may include various data (or information) collected by sensor 164. In an example, the SENS may include data collected in response to motion detected in the monitored field of view, ambient light levels in the monitored field of view, and / or sound picked up in the monitored field of view. However, other types of data may be collected and / or generated based on application-specific design criteria. The SENS may be presented to processor / SoC 102. In an example, sensor 164 may generate (assert) the SENS when motion is detected in the field of view monitored by camera system 100. In another example, sensor 164 may generate (assert) the SENS when audio is triggered in the field of view monitored by camera system 100. In yet another example, sensor 164 may be configured to provide directional information about motion and / or sound detected in the field of view. This directional information may also be transmitted to processor / SoC 102 via the SENS.

[0069] HID 166 can implement an input device. For example, HID 166 can be configured to receive human input. In one example, HID 166 can be configured to receive password input from a user. In another example, HID 166 can be configured to receive user input to provide various parameters and / or settings to processor 102 and / or memory 150. In some embodiments, camera system 100 may include a keyboard, touchpad (or screen), doorbell switch, and / or other human-machine interface device (HID) 166. In an example, sensor 164 can be configured to determine when an object approaches HID 166. In an example where camera system 100 is implemented as part of an access control application, capture device 104 can be activated to provide images for identifying a person attempting access, and a lock area and / or illumination for access touchpad 166 can be activated. For example, a combination of input from HID 166 (e.g., a password or PIN code) can be combined with activity determination and / or depth analysis performed by processor 102 to achieve two-factor authentication.

[0070] The processor / SoC 102 can receive a signal VIDEO and a signal SENS. The processor / SoC 102 can generate one or more video output signals (e.g., VIDOUT), one or more control signals (e.g., CTRL), and / or one or more depth data signals (e.g., DIMAGES) based on the signal VIDEO, signal SENS, and / or other inputs. In some embodiments, the signals VIDOUT, DIMAGES, and CTRL can be generated based on analysis of the signal VIDEO and / or objects detected in the signal VIDEO.

[0071] In various embodiments, the processor / SoC 102 may be configured to perform one or more of the following: feature extraction, object detection, object tracking, 3D reconstruction, liveness detection, and object recognition. For example, the processor / SoC 102 may determine motion information and / or depth information by analyzing frames from a signal VIDEO and comparing those frames with previous frames. The comparison may be used to perform digital motion estimation. In some embodiments, the processor / SoC 102 may be configured to generate a video output signal VIDOUT including video data and / or a depth data signal DIMAGES including a disparity map and a depth map from the signal VIDEO. The video output signal VIDOUT and / or the depth data signal DIMAGES may be presented to memory 150, communication module 154, and / or wireless interface 156. In some embodiments, the video signal VIDOUT and / or the depth data signal DIMAGES may be used internally by the processor 102 (e.g., not presented as output).

[0072] The signal VIDOUT can be presented to the communication device 156. In some embodiments, the signal VIDOUT may include encoded video frames generated by the processor 102. In some embodiments, the encoded video frames may include a complete video stream (e.g., encoded video frames representing all video captured by the capture device 104). The encoded video frames may be encoded, cropped, stitched, and / or enhanced versions of pixel data received from the signal VIDEO. In the example, the encoded video frames may be high-resolution, digital, encoded, de-distorted, stabilized, cropped, blended, stitched, and / or rolling shutter effect corrected versions of the signal VIDEO.

[0073] In some embodiments, the signal VIDOUT may be generated based on video analysis (e.g., computer vision operations) performed by processor 102 on generated video frames. Processor 102 may be configured to perform computer vision operations to detect objects and / or events in the video frames, and then convert the detected objects and / or events into statistical data and / or parameters. In one example, the data determined by the computer vision operations may be converted by processor 102 into a human-readable format. Data from the computer vision operations can be used to detect objects and / or events. The computer vision operations may be performed locally by processor 102 (e.g., without needing to communicate with external devices to offload computational operations). For example, locally executed computer vision operations allow the computer vision operations to be performed by processor 102 and avoid heavy video processing running on a backend server. Avoiding video processing on a backend (e.g., remote location) server can protect privacy.

[0074] In some embodiments, the signal VIDOUT can be data generated by processor 102 (e.g., video analysis results, audio / speech analysis results, etc.) that can be transmitted to a cloud computing service for information aggregation and / or to provide training data for machine learning (e.g., to improve object detection, improve audio detection, improve liveness detection, etc.). In some embodiments, the signal VIDOUT can be provided to a cloud service for mass storage (e.g., to enable users to retrieve encoded video using smartphones and / or desktop computers). In some embodiments, the signal VIDOUT can include data extracted from video frames (e.g., computer vision results) and can transmit the results to another device (e.g., a remote server, a cloud computing system, etc.) to offload the analysis of the results to another device (e.g., offloading the analysis of the results to a cloud computing service instead of performing all the analysis locally). The type of information transmitted by the signal VIDOUT can vary depending on the design criteria of a particular implementation.

[0075] The CTRL signal can be configured to provide a control signal. The CTRL signal can be generated in response to a decision made by processor 102. In an example, the CTRL signal can be generated in response to a detected object and / or features extracted from a video frame. The CTRL signal can be configured to enable, disable, or change the operating mode of another device. In one example, the CTRL signal can be used to lock / unlock a door controlled by an electronic lock. In another example, the device can be set to sleep mode (e.g., low power mode) and / or activated from sleep mode in response to the CTRL signal. In yet another example, the CTRL signal can be used to generate an alarm and / or notification. The type of device controlled by the CTRL signal and / or the response performed by the device in response to the CTRL signal can vary depending on the design criteria of the specific implementation.

[0076] The signal CTRL can be generated based on data received by sensor 164 (e.g., temperature readings, motion sensor readings, etc.). The signal CTRL can be generated based on input from HID 166. The signal CTRL can be generated based on human behavior detected by processor 102 in a video frame. The signal CTRL can be generated based on the type of detected object (e.g., human, animal, vehicle, etc.). The signal CTRL can be generated in response to the detection of a specific type of object at a specific location. The signal CTRL can be generated in response to the detection of a specific type of object at a specific location. The signal CTRL can be generated in response to user input to provide various parameters and / or settings to processor 102 and / or memory 150. Processor 102 can be configured to generate the signal CTRL in response to sensor fusion operations (e.g., aggregation of information received from different sources). Processor 102 can be configured to generate the signal CTRL in response to the result of a liveness detection performed by processor 102. The conditions used to generate the signal CTRL can vary depending on the design criteria of the specific implementation.

[0077] Signals DIMAGES may include one or more depth maps and / or disparity maps generated by processor 102. Signals DIMAGES may be generated in response to 3D reconstruction performed on a monocular single-channel image. Signals DIMAGES may be generated in response to analysis of captured video data and structured light pattern SLP.

[0078] A multi-step approach can be implemented to activate and / or disable the capture device 104 based on the output of the motion sensor 164 and / or any other power consumption characteristics of the camera system 100, thereby reducing the power consumption of the camera system 100 and extending the lifespan of the battery 152. The motion sensor in sensor 164 can have low power consumption on the battery 152 (e.g., less than 10W). In the example, the motion sensor of sensor 164 can be configured to remain on (e.g., always active) unless disabled in response to feedback from the processor / SoC 102. Video analysis performed by the processor / SoC 102 may have relatively high power consumption on the battery 152 (e.g., greater than that of the motion sensor 164). In the example, the processor / SoC 102 can be in a low-power state (or powered off) until some motion is detected by the motion sensor of sensor 164.

[0079] The camera system 100 can be configured to operate using various power states. For example, in a power-off state (e.g., sleep state, low power state), sensor 164 and the motion sensor of processor / SoC 102 can be turned on, and other components of the camera system 100 (e.g., image capture device 104, memory 150, communication module 154, etc.) can be turned off. In another example, the camera system 100 can operate in an intermediate state. In the intermediate state, image capture device 104 can be turned on, and memory 150 and / or communication module 154 can be turned off. In yet another example, the camera system 100 can operate in a powered-on (or high-power) state. In the powered-on state, sensor 164, processor / SoC 102, capture device 104, memory 150, and / or communication module 154 can be turned on. The camera system 100 can consume some power from battery 152 (e.g., a relatively small and / or minimal amount of power) in the power-off state. In the powered-on state, the camera system 100 can consume more power from battery 152. The number of power states and / or the number of components of the camera system 100 that are turned on when the camera system 100 operates in each power state can vary according to the design criteria of a particular implementation.

[0080] In some embodiments, the camera system 100 may be implemented as a system-on-a-chip (SoC). For example, the camera system 100 may be implemented as a printed circuit board including one or more components. The camera system 100 may be configured to perform intelligent video analysis on video frames of a video. The camera system 100 may be configured to crop and / or enhance the video.

[0081] In some embodiments, the video frame may be a view (or a derivative of a view) captured by the capture device 104. Pixel data signals may be enhanced by the processor 102 (e.g., color conversion, noise filtering, automatic exposure, automatic white balance, automatic focus, etc.). In some embodiments, the video frame may provide a series of cropped and / or enhanced video frames that improve the view from the perspective of the camera system 100 (e.g., providing night vision, providing high dynamic range (HDR) imaging, providing more viewing area, highlighting detected objects, providing additional data (e.g., digital distance to the detected object), etc.) to enable the processor 102 to see the location better than a human can see with human vision.

[0082] Encoded video frames can be processed locally. In one example, the encoded video can be stored locally by memory 150 so that processor 102 can facilitate computer vision analysis internally (e.g., without first uploading the video frames to a cloud service). Processor 102 can be configured to select video frames to be encapsulated into a video stream that can be transmitted over a network (e.g., a bandwidth-limited network).

[0083] In some embodiments, processor 102 may be configured to perform sensor fusion operations. The sensor fusion operations performed by processor 102 may be configured to analyze information from multiple sources (e.g., capture device 104, sensor 164, and HID 166). By analyzing various data from different sources, the sensor fusion operations may be able to make inferences about the data that might not be possible from just one data source. For example, the sensor fusion operations implemented by processor 102 may analyze video data (e.g., human mouth movements) and speech patterns from directional audio. Different sources can be used to develop scene models to support decision-making. For example, processor 102 may be configured to compare the synchronization of detected speech patterns with mouth movements in video frames to determine which person is speaking in the video frame. The sensor fusion operations may also provide temporal correlation, spatial correlation, and / or reliability of the received data.

[0084] In some embodiments, processor 102 may implement convolutional neural network (CNN) capabilities. CNN capabilities can be implemented using deep learning techniques for computer vision. CNN capabilities can be configured to perform pattern and / or image recognition using a training process involving multi-layer feature detection. Computer vision and / or CNN capabilities can be executed locally by processor 102. In some embodiments, processor 102 may receive training data and / or feature set information from external sources. For example, external devices (e.g., cloud services) may access various data sources to provide training data that the camera system 100 may not be able to obtain. However, computer vision operations performed using feature sets can be performed using the computational resources of processor 102 within the camera system 100.

[0085] The video pipeline of processor 102 can be configured to perform local dedistortion, cropping, enhancement, rolling shutter correction, stabilization, downsizing, packing, compression, conversion, mixing, synchronization, and / or other video operations. The video pipeline of processor 102 can enable multi-stream support (e.g., generating multiple bitstreams in parallel, each including a different bitrate). In the example, the video pipeline of processor 102 can implement an image signal processor (ISP) with an input pixel rate of 320 Mbps. The architecture of the video pipeline of processor 102 enables real-time and / or near real-time video operations on high-resolution video and / or high-bitrate video data. The video pipeline of processor 102 can implement computer vision processing, stereo vision processing, object detection, 3D noise reduction, fisheye lens correction (e.g., real-time 360-degree dedistortion and lens distortion correction), oversampling, and / or high dynamic range processing on 4K resolution video data. In one example, the video pipeline architecture can achieve 4K ultra-high resolution with H.264 encoding at dual real-time rates (e.g., 60fps) and 4K ultra-high resolution with H.265 / HEVC and / or 4KAVC encoding at 30fps (e.g., multi-stream 4KP30AVC and HEVC encoding). The type of video operation and / or the type of video data operated on by the processor 102 can vary depending on the design criteria of the specific implementation.

[0086] The camera sensor 180 can be a high-resolution sensor. Using the high-resolution sensor 180, the processor 102 can combine oversampling of the image sensor 180 with digital scaling within the cropped area. Each of oversampling and digital scaling can be one of the video operations performed by the processor 102. Oversampling and digital scaling can be implemented to provide a higher resolution image within the overall size constraints of the cropped area.

[0087] In some embodiments, lens 160 may be a fisheye lens. One of the video operations performed by processor 102 may be a de-distortion operation. Processor 102 may be configured to de-distort the generated video frames. De-distortion may be configured to reduce and / or remove severe distortion caused by fisheye lens and / or other lens characteristics. For example, de-distortion may reduce and / or eliminate bulging effects to provide a linear image.

[0088] Processor 102 can be configured to crop (e.g., trim) a region of interest from a full video frame (e.g., generate a region of interest video frame). Processor 102 can generate video frames and select regions. In the example, cropping the region of interest can generate a second image. The cropped image (e.g., the region of interest video frame) can be smaller than the original video frame (e.g., the cropped image can be a portion of the captured video).

[0089] The region of interest (ROI) can be dynamically adjusted based on the location of the audio source. For example, the detected audio source may be moving, and its location may shift as video frames are captured. Processor 102 can update the coordinates of the selected ROI and dynamically update the cropped portion (e.g., a directional microphone implemented as one or more sensors in sensor 164 can dynamically update its position based on captured directional audio). The cropped portion may correspond to the selected ROI. As the ROI changes, the cropped portion may change. For example, the selected coordinates of the ROI may change frame by frame, and processor 102 may be configured to crop the selected region in each frame.

[0090] Processor 102 can be configured to oversample image sensor 180. Oversampling of image sensor 180 can produce a higher resolution image. Processor 102 can also be configured to digitally magnify regions of video frames. For example, processor 102 can digitally magnify a cropped region of interest. For example, processor 102 can establish a region of interest based on directional audio, crop the region of interest, and then digitally magnify the cropped region of interest video frame.

[0091] The de-distortion operation performed by processor 102 can adjust the visual content of the video data. The adjustment performed by processor 102 can make the visual content look natural (e.g., look as if it were seen by a person viewing a position corresponding to the field of view of capture device 104). In the example, de-distortion can alter the video data to generate linear video frames (e.g., correcting artifacts caused by lens characteristics of lens 160). De-distortion operations can be implemented to correct distortions caused by lens 160. Adjusted visual content can be generated to achieve more accurate and / or reliable object detection.

[0092] Various features (e.g., de-distortion, digital scaling, cropping, etc.) can be implemented as hardware modules in processor 102. Implementing hardware modules can increase the video processing speed of processor 102 (e.g., faster than software implementation). Hardware implementation enables video to be processed while reducing latency. The hardware components used can vary depending on the design standards of the specific implementation.

[0093] Processor 102 is shown as including multiple blocks (or circuits) 190a-190n. Blocks 190a-190n can implement various hardware modules implemented by processor 102. Hardware modules 190a-190n can be configured to provide various hardware components to implement a video processing pipeline. Circuits 190a-190n can be configured to receive pixel data VIDEO, generate video frames from the pixel data, perform various operations on the video frames (e.g., de-distortion, rolling shutter correction, cropping, magnification, image stabilization, 3D reconstruction, liveness detection, etc.), prepare video frames for communication with external hardware (e.g., encoding, encapsulation, color correction, etc.), parse feature sets, and implement various operations for computer vision (e.g., object detection, segmentation, classification, etc.). Hardware modules 190a-190n can be configured to implement various security features (e.g., secure boot, I / O virtualization, etc.). Various implementations of processor 102 may not necessarily utilize all features of hardware modules 190a-190n. The features and / or functions of hardware modules 190a-190n may vary depending on the design criteria of a particular implementation. Details of hardware modules 190a-190n can be described in conjunction with U.S. Patent Application No. 16 / 831,549, filed April 16, 2020; U.S. Patent Application No. 16 / 288,922, filed February 28, 2019; U.S. Patent Application No. 15 / 593,493 (now U.S. Patent No. 10,437,600), filed May 12, 2017; U.S. Patent Application No. 15 / 931,942, filed May 14, 2020; U.S. Patent Application No. 16 / 991,344, filed August 12, 2020; and U.S. Patent Application No. 17 / 479,034, filed September 20, 2021, the appropriate portions of which are incorporated herein by reference in their entirety.

[0094] Hardware modules 190a-190n can be implemented as dedicated hardware modules. Compared to software implementation, using dedicated hardware modules 190a-190n to implement various functions of processor 102 allows processor 102 to be highly optimized and / or customized to limit power consumption, reduce heat generation, and / or increase processing speed. Hardware modules 190a-190n can be customizable and / or programmable to implement multiple types of operations. Implementing dedicated hardware modules 190a-190n allows the hardware used to perform each type of computation to be optimized for speed and / or efficiency. For example, hardware modules 190a-190n can implement several relatively simple operations frequently used in computer vision operations, which together enable computer vision operations to be performed in real time. The video pipeline can be configured to: identify objects. Objects can be identified by interpreting numerical and / or symbolic information to determine that visual data represents a specific type of object and / or feature. For example, the number of pixels and / or pixel color of video data can be used to identify portions of the video data as objects. Hardware modules 190a-190n enable computationally intensive operations (e.g., computer vision operations, video encoding, video transcoding, 3D reconstruction, depth map generation, liveness detection, etc.) to be performed locally by the camera system 100.

[0095] One of the hardware modules 190a-190n (e.g., 190a) can implement a scheduler circuit. Scheduler circuit 190a can be configured to store a directed acyclic graph (DAG). In the example, scheduler circuit 190a can be configured to generate and store a DAG in response to received (e.g., loaded) feature set information. The DAG can define video operations to be performed to extract data from video frames. For example, the DAG can define various mathematical weights (e.g., neural network weights and / or biases) to be applied when performing computer vision operations to classify various groups of pixels into specific objects.

[0096] Scheduler circuit 190a can be configured to parse acyclic graphs to generate various operators. Operators can be scheduled by scheduler circuit 190a in one or more hardware modules 190a-190n. For example, one or more hardware modules 190a-190n can implement a hardware engine configured to perform a specific task (e.g., a hardware engine designed to perform repetitive specific mathematical operations used for performing computer vision operations). Scheduler circuit 190a can schedule operators based on when they are ready to be processed by hardware engines 190a-190n.

[0097] Scheduler circuit 190a can multiplex task time across hardware modules 190a-190n based on their availability. Scheduler circuit 190a can parse a directed acyclic graph (DAG) into one or more data streams. Each data stream can include one or more operators. Once the DAG is parsed, scheduler circuit 190a can assign data streams / operators to hardware engines 190a-190n and send relevant operator configuration information to initiate the operators.

[0098] Each directed acyclic sphere Figure Two The radix representation can be an ordered traversal of a directed acyclic graph, where descriptors and operators are intertwined based on data dependencies. Descriptors typically provide registers that link data buffers to specific operands in the associated operators. In various embodiments, operators may not appear in the directed acyclic graph representation until all associated descriptors have been declared for operands.

[0099] One of the hardware modules 190a-190n (e.g., 190b) can implement an Artificial Neural Network (ANN) module. The ANN module can be implemented as a fully connected neural network or a Convolutional Neural Network (CNN). In this example, the fully connected network is "structure-agnostic" because no special assumptions need to be made about the input. A fully connected neural network consists of a series of fully connected layers that connect each neuron in one layer to each neuron in another layer. In a fully connected layer, there are n*m ​​weights for n inputs and m outputs. Each output node also has a bias value, resulting in a total of (n+1)*m parameters. In a trained neural network, (n+1)*m parameters have been determined during the training process. A trained neural network typically includes an architectural specification and a set of parameters (weights and biases) determined during the training process. In another example, a CNN architecture may explicitly assume that the input is an image in order to be able to encode specific attributes into the model architecture. A CNN architecture may include a sequence of layers, each layer transforming one activation into another through a differentiable function.

[0100] In the example shown, the artificial neural network 190b can implement a convolutional neural network (CNN) module. The CNN module 190b can be configured to perform computer vision operations on video frames. The CNN module 190b can be configured to recognize objects through multi-layer feature detection. The CNN module 190b can be configured to compute descriptors based on the performed feature detections. The descriptors enable the processor 102 to determine the probability that a pixel in a video frame corresponds to a specific object (e.g., a specific brand / model / year of a vehicle, identifying a person as a specific individual, detecting animal types, detecting facial features, etc.).

[0101] CNN module 190b can be configured to: implement convolutional neural network capabilities; implement computer vision using deep learning techniques; implement pattern and / or image recognition using a training process involving multi-layer feature detection; and perform inference against machine learning models.

[0102] CNN module 190b can be configured to perform feature extraction and / or matching solely in hardware. Feature points typically represent regions of interest (e.g., corners, edges, etc.) within a video frame. By tracking feature points over time, estimates of the self-motion of the capture platform or motion models of observed objects in the scene can be generated. To track feature points, the matching operation is typically incorporated into CNN module 190b by hardware to find the most probable correspondence between feature points in a reference and target video frame. During the matching of reference and target feature point pairs, each feature point can be represented by a descriptor (e.g., image patch, SIFT, BRIEF, ORB, FREAK, etc.). Implementing CNN module 190b using dedicated hardware circuitry enables real-time computation of descriptor matching distances.

[0103] The CNN module 190b can be configured to perform face detection, face recognition, and / or liveness detection. For example, face detection, face recognition, and / or liveness detection can be performed based on a trained neural network implemented by the CNN module 190b. In some embodiments, the CNN module 190b can be configured to generate a depth map from a structured light pattern. The CNN module 190b can be configured to perform various detection and / or recognition operations and / or perform 3D recognition operations.

[0104] CNN module 190b can be a dedicated hardware module configured to perform feature detection of video frames. Features detected by CNN module 190b can be used to compute descriptors. CNN module 190b can determine the probability that a pixel in a video frame belongs to a specific object and / or some objects in response to the descriptors. For example, using the descriptors, CNN module 190b can determine the probability that a pixel corresponds to a specific object (e.g., a person, a piece of furniture, a pet, a vehicle, etc.) and / or characteristics of the object (e.g., the shape of the eyes, the distance between facial features, a vehicle hood, body parts, a vehicle license plate, a face, clothing worn by a person, etc.). Implementing CNN module 190b as a dedicated hardware module of processor 102 enables device 100 to perform computer vision operations locally (e.g., on-chip) without relying on the processing power of a remote device (e.g., transmitting data to a cloud computing service).

[0105] The computer vision operations performed by the CNN module 190b can be configured to: perform feature detection on video frames to generate descriptors. The CNN module 190b can perform object detection to determine regions in the video frames with a high probability of matching a specific object. In one example, the type of object to be matched (e.g., a reference object) can be customized using an open operand stack (enabling the programmability of the processor 102 to implement various artificial neural networks defined by directed acyclic graphs, each providing instructions for performing various types of object detection). The CNN module 190b can be configured to: perform local masking on regions with a high probability of matching a specific object to detect the object.

[0106] In some embodiments, the CNN module 190b can determine the location (e.g., 3D coordinates and / or position coordinates) of various features (e.g., characteristics) of the detected object. In one example, 3D coordinates can be used to determine the position of a person's arms, legs, chest, and / or eyes. A position coordinate on a first axis representing the vertical position of a body part in 3D space and another coordinate on a second axis representing the horizontal position of a body part in 3D space can be stored. In some embodiments, the distance from the lens 160 can represent a coordinate of the depth position of the body part in 3D space (e.g., a position coordinate on a third axis). Using the various body parts' positions in 3D space, the processor 102 can determine the detected person's body position and / or body characteristics.

[0107] The CNN module 190b can be pre-trained (e.g., configured to perform computer vision to detect objects based on training data received to train the CNN module 190b). For example, the results of the training data (e.g., a machine learning model) can be pre-programmed and / or loaded into processor 102. The CNN module 190b can perform inference on the machine learning model (e.g., to perform object detection). Training can include determining weight values ​​for each layer of the neural network model. For example, weight values ​​can be determined for each layer for feature extraction (e.g., convolutional layers) and / or for classification (e.g., fully connected layers). The weight values ​​learned by the CNN module 190b can vary depending on the design criteria of a particular implementation.

[0108] The CNN module 190b can perform feature extraction and / or object detection by performing convolution operations. These convolution operations can be hardware-accelerated for fast (e.g., real-time) computation that can be performed with low power consumption. In some embodiments, the convolution operations performed by the CNN module 190b can be used to perform computer vision operations. In some embodiments, the convolution operations performed by the CNN module 190b can be used for any function (e.g., 3D reconstruction) that may involve computing convolution operations and is performed by the processor 102.

[0109] Convolution operations can include sliding a feature detection window along a layer while performing computations (e.g., matrix operations). The feature detection window can apply filters to pixels and / or extract features associated with each layer. The feature detection window can be applied to a single pixel and multiple surrounding pixels. In the example, these layers can be represented as matrices representing the values ​​of pixels and / or features of one of these layers, and the filters applied by the feature detection window can be represented as matrices. Convolution operations can apply matrix multiplication between regions of the current layer covered by the feature detection window. Convolution operations can slide the feature detection window along the region of the layer to generate a result representing each region. The size of the regions, the type of operation for applying filters, and / or the number of layers can vary depending on the design criteria of a particular implementation.

[0110] Using convolutional operations, the CNN module 190b can compute multiple features of pixels in the input image at each extraction step. For example, each layer can receive input from a set of features located in a small neighborhood (e.g., a region) of the previous layer (e.g., a local receptive field). Convolutional operations can extract basic visual features (e.g., oriented edges, endpoints, corners, etc.), which are then combined by higher layers. Because the feature extraction window operates on pixels and their nearby pixels (or subpixels), the results of the operations may be position-invariant. These layers can include convolutional layers, pooling layers, non-linear layers, and / or fully connected layers. In the example, convolutional operations can learn to detect edges from raw pixels (e.g., the first layer), then use features from the previous layer (e.g., detected edges) to detect shapes in the next layer, and then use these shapes to detect higher-level features in higher layers (e.g., facial features, pets, vehicles, vehicle parts, furniture, etc.), and the final layer may be a classifier using higher-level features.

[0111] The CNN module 190b can perform data stream operations for feature extraction and matching, including two-stage detection, transformation operators, component operators manipulating component lists (e.g., components can be regions of vectors sharing common attributes and can be combined with bounding boxes), matrix inversion operators, dot product operators, convolution operators, conditional operators (e.g., multiplexing and demultiplexing), remapping operators, min-max-reduction operators, pooling operators, non-minimum and non-maximum suppression operators, non-maximum suppression operators based on scanning windows, aggregation operators, dispersion operators, statistical operators, classification operators, integral image operators, comparison operators, indexing operators, pattern matching operators, feature extraction operators, feature detection operators, two-stage object detection operators, score generation operators, block reduction operators, and upsampling operators. The types of operations performed by the CNN module 190b for extracting features from training data can vary depending on the design criteria of a particular implementation.

[0112] Each of the hardware modules 190a-190n can implement a processing resource (or hardware resource or hardware engine). Hardware engines 190a-190n can be used to perform specific processing tasks. In some configurations, hardware engines 190a-190n can operate in parallel and independently of each other. In other configurations, hardware engines 190a-190n can operate collaboratively with each other to perform assigned tasks. One or more hardware engines among the hardware engines 190a-190n can be homogeneous processing resources (all circuits 190a-190n can have the same capabilities) or heterogeneous processing resources (two or more circuits 190a-190n can have different capabilities).

[0113] refer to Figure 4 The diagram illustrates the processing circuitry of a camera system 100 configured to perform 3D reconstruction using a convolutional neural network. In this example, the processing circuitry of the camera system 100 can be configured for a variety of applications, including but not limited to: autonomous and semi-autonomous vehicles (e.g., cars, trucks, motorcycles, agricultural machinery, drones, aircraft, etc.), manufacturing, and / or security and surveillance systems. Compared to a general-purpose computer, the processing circuitry of the camera system 100 typically includes hardware circuitry optimized to provide high-performance image processing and computer vision pipelines with minimal area and power consumption. In this example, various operations for performing image processing, feature detection / extraction, 3D reconstruction, liveness detection, depth map generation, and / or object detection / classification for computer (or machine) vision can be implemented using hardware modules designed to reduce computational complexity and utilize resources efficiently.

[0114] In an example embodiment, processing circuitry 100 may include processor 102, memory 150, general-purpose processor 158, and / or memory bus 200. General-purpose processor 158 may implement a first processor. Processor 102 may implement a second processor. In this example, circuitry 102 may implement a computer vision processor. In this example, processor 102 may be an intelligent vision processor. Memory 150 may implement external memory (e.g., memory outside of circuitry 158 and 102). In this example, circuitry 150 may be implemented as dynamic random access memory (DRAM) circuitry. The processing circuitry of camera system 100 may include other components (not shown). The number, type, and / or arrangement of components in the processing circuitry of camera system 100 may vary depending on the design criteria of a particular implementation.

[0115] A general-purpose processor 158 can operate to interact with circuits 102 and 150 to perform various processing tasks. In an example, processor 158 can be configured as a controller for circuit 102. Processor 158 can be configured to execute computer-readable instructions. In one example, the computer-readable instructions may be stored by circuit 150. In some embodiments, the computer-readable instructions may include controller operations. Processor 158 can be configured to communicate with circuit 102 and / or access results produced by components of circuit 102. In an example, processor 158 can be configured to utilize circuit 102 to perform operations associated with one or more neural network models.

[0116] In the example, processor 102 typically includes scheduler circuitry 190a, block (or circuitry) 202, one or more blocks (or circuitries) 204a-204n, block (or circuitry) 206, and path 208. Block 202 may implement a directed acyclic graph (DAG) memory. DAG memory 202 may include CNN module 190b and / or weights / biases 210. Blocks 204a-204n may implement hardware resources (or engines). Block 206 may implement shared memory circuitry. In the example embodiment, one or more circuits in circuits 204a-204n may include blocks (or circuitries) 212a-212n. In the illustrated example, circuits 212a and 212b are implemented as representative examples in the corresponding hardware engines 204a-204b. One or more circuits in circuits 202, 204a-204n, and / or circuit 206 may be related to... Figure 3 Example implementations of hardware modules 190a-190n are shown in association.

[0117] In the example, processor 158 may be configured to program circuit 102 using one or more pre-trained artificial neural network models (ANNs), including a convolutional neural network (CNN) 190b having multiple output frames according to embodiments of the invention and weights / kernels (WGTS) 210 used by the CNN module 190b. In various embodiments, the CNN module 190b may be configured (trained) for operation in an edge device. In the example, the processing circuitry of camera system 100 may be coupled to a sensor (e.g., a video camera, etc.) configured to generate data input. The processing circuitry of camera system 100 may be configured to generate one or more outputs in response to data input from the sensor, based on one or more inferences made by the pre-trained CNN module 190b using weights / kernels (WGTS) 210. The operations performed by processor 158 may vary depending on the design criteria of a particular implementation.

[0118] In various embodiments, circuit 150 may implement dynamic random access memory (DRAM) circuitry. Circuit 150 is typically operable as a multidimensional array for storing input data elements and various forms of output data elements. Circuit 150 may exchange input data elements and output data elements with processor 158 and processor 102.

[0119] Processor 102 may implement computer vision processor circuitry. In examples, processor 102 may be configured to implement various functions for computer vision. Processor 102 is generally operable to perform specific processing tasks arranged by processor 158. In various embodiments, all or part of processor 102 may be implemented individually in hardware. Processor 102 may directly execute data streams involving the execution of CNN module 190b and generated by software (e.g., directed acyclic graphs, etc.) for specified processing tasks (e.g., computer vision, 3D reconstruction, liveness detection, etc.). In some embodiments, processor 102 may be a representative example of a multitude of computer vision processors implemented by the processing circuitry of camera system 100 and configured to operate together.

[0120] In one example, circuit 212a can implement a convolution operation. In another example, circuit 212b can be configured to provide a dot product operation. Convolution and dot product operations can be used to perform computer (or machine) vision tasks (e.g., as part of an object detection process). In yet another example, one or more of the circuits 204c-204n may include blocks (or circuits) 212c-212n (not shown) for providing multidimensional convolution computation. In yet another example, one or more of the circuits 204a-204n can be configured to perform a 3D reconstruction task.

[0121] In the example, circuit 102 can be configured to receive a directed acyclic graph (DAG) from processor 158. The DAG received from processor 158 can be stored in DAG memory 202. Circuit 102 can be configured to perform the DAG for CNN module 190b using circuits 190a, 204a-204n, and 206.

[0122] Multiple signals (e.g., OP_A-OP_N) can be exchanged between circuit 190a and corresponding circuits 204a-204n. Each signal in OP_A-OP_N can convey execution operation information and / or concession operation information. Multiple signals (e.g., MEM_A-MEM_N) can be exchanged between the respective circuits 204a-204n and circuit 206. Signals MEM_A-MEM_N can carry data. Signals (e.g., DRAM) can be exchanged between circuit 150 and circuit 206. Signal DRAM can transmit data between circuits 150 and 190a (e.g., on transmission path 208).

[0123] Scheduler circuit 190a is typically operable to schedule tasks within circuits 204a-204n to perform various computer vision-related tasks defined by processor 158. Individual tasks can be assigned by scheduler circuit 190a to circuits 204a-204n. Scheduler circuit 190a can assign individual tasks in response to parsing a directed acyclic graph (DAG) provided by processor 158. Scheduler circuit 190a can time-multiplex tasks across circuits 204a-204n based on their availability for performing work.

[0124] Each circuit 204a-204n can implement a processing resource (or hardware engine). Hardware engines 204a-204n are typically operable to perform a specific processing task. Hardware engines 204a-204n can be implemented as including dedicated hardware circuitry optimized for high performance and low power consumption while performing a specific processing task. In some configurations, hardware engines 204a-204n can operate in parallel and independently of each other. In other configurations, hardware engines 204a-204n can operate collaboratively with each other to perform assigned tasks.

[0125] Hardware engines 204a-204n can be homogeneous processing resources (e.g., all circuits 204a-204n can have the same capabilities) or heterogeneous processing resources (e.g., two or more circuits 204a-204n can have different capabilities). Hardware engines 204a-204n are typically configured to execute operators, which may include, but are not limited to, resampling operators, transformation operators, component operators manipulating a list of components (e.g., components can be vector regions sharing common attributes and can be grouped together with bounding boxes), matrix inversion operators, dot product operators, convolution operators, conditional operators (e.g., multiplexing and demultiplexing), remapping operators, minimum-maximum-reduction operators, pooling operators, non-minimum, non-maximum suppression operators, aggregation operators, dispersion operators, statistical operators, classification operators, integral image operators, upsampling operators, and powers of two downsampling operators, etc.

[0126] In the example, hardware engines 204a-204n may include matrices stored in various memory buffers. The matrices stored in the memory buffers can be used to initialize convolution operators. The convolution operators can be configured to efficiently perform computations that are repeatedly executed on a convolution function. In the example, hardware engines 204a-204n implementing the convolution operators may include multiple mathematical circuits configured to process multi-bit input values ​​and operate in parallel. Convolution operators can provide an efficient and general solution for computer vision and / or 3D reconstruction by using one-dimensional or higher-dimensional kernels to compute convolutions (also known as cross-correlation). Convolution can be used for computer vision operations such as object detection, object recognition, edge enhancement, image smoothing, etc. The techniques and / or architectures implemented by this invention are operable for computing convolutions of input arrays with kernels. Details of the convolution operators can be described in conjunction with U.S. Patent No. 10,310,768, filed January 11, 2017, the appropriate portions of which are incorporated herein by reference.

[0127] In various embodiments, hardware engines 204a-204n can be implemented as individual hardware circuits. In some embodiments, hardware engines 204a-204n can be implemented as general-purpose engines that can be configured to operate as dedicated machines (or engines) through circuit customization and / or software / firmware. In some embodiments, hardware engines 204a-204n can alternatively be implemented as one or more instances or threads of program code executing on processor 158 and / or one or more processors 102, including but not limited to vector processors, central processing units (CPUs), digital signal processors (DSPs), or graphics processing units (GPUs). In some embodiments, scheduler 190a can select one or more of hardware engines 204a-204n for a specific process and / or thread. Scheduler 190a can be configured to assign hardware engines 204a-204n to a specific task in response to parsing a directed acyclic graph stored in DAG memory 202.

[0128] Circuit 206 can implement shared memory circuitry. Shared memory 206 can be configured to store data in response to input requests and / or present data in response to output requests (e.g., requests from processor 158, DRAM 150, scheduler circuitry 190a, and / or hardware engines 204a-204n). In this example, shared memory circuitry 206 can implement on-chip memory for computer vision processor 102. Shared memory 206 is generally operable to store all or part of a multidimensional array (or vector) of input and output data elements generated and / or utilized by hardware engines 204a-204n. Input data elements can be transferred from DRAM circuitry 150 to shared memory 206 via memory bus 200. Output data elements can be sent from shared memory 206 to DRAM circuitry 150 via memory bus 200.

[0129] Path 208 can implement a transfer path within processor 102. Transfer path 208 is typically operable to move data from scheduler circuit 190a to shared memory 206. Transfer path 208 can also be operable to move data from shared memory 206 to scheduler circuit 190a.

[0130] Processor 158 is shown communicating with computer vision processor 102. Processor 158 may be configured as a controller of computer vision processor 102. In some embodiments, processor 158 may be configured to transmit instructions to scheduler 190a. For example, processor 158 may provide one or more directed acyclic graphs (DAGs) to scheduler 190a via DAG memory 202. Scheduler 190a may initialize and / or configure hardware engines 204a-204n in response to parsing the DAGs. In some embodiments, processor 158 may receive status information from scheduler 190a. For example, scheduler 190a may provide processor 158 with status information and / or readiness status from the outputs of hardware engines 204a-204n, enabling processor 158 to determine one or more next instructions to execute and / or decisions to make. In some embodiments, processor 158 may be configured to communicate with shared memory 206 (e.g., directly or via scheduler 190a, which receives data from shared memory 206 via path 208). Processor 158 can be configured to retrieve information from shared memory 206 to make a decision. The instructions executed by processor 158 in response to information from computer vision processor 102 can vary depending on the design criteria of a particular implementation.

[0131] refer to Figure 5 , Figure 5A diagram illustrating example video frames including large and small facial regions is shown. Example video frame 250 is shown. Example video frame 250 can be an illustrative example of a video frame in a sequence of video frames analyzed by processor 102. Example video frame 250 can include pixel data of the video frame arranged to be provided to CNN module 190b. CNN module 190b can be configured to analyze the pixel data of video frame 250 to detect the size and / or location of facial regions of a person captured in video frame 250. In the example, video frame 250 can include an RGB image captured when structured light projector 106 is not projecting a structured light pattern SLP.

[0132] Example video frame 250 may be video captured at the entrance to a secure area (e.g., a view from a security device implementing capture device 104 to allow access through a locked door). Example video frame 250 may be video captured by a smartphone to unlock access to the smartphone. Example video frame 250 may be video captured by a smart doorbell and / or a home security camera. Example video frame 250 may be video captured by a camera configured to focus on a detected face. The use cases for capturing and analyzing example video frame 250 may vary depending on the design criteria of a particular implementation.

[0133] Example video frame 250 may include person 252 and person 254. Person 252's face 260 is shown occupying most of example video frame 250. For example, face 260 may be in the foreground of example video frame 250 (e.g., in front of capture device 104) close to image sensor 180. Person 254's face 262 is shown occupying a small portion of example video frame 250. For example, face 262 may be further away from image sensor 180 than face 260. Face 262 may be in the background of example video frame 250. In one example, person 252 may be an object authorized by device 100, and person 254 may be a pedestrian (e.g., a bystander who may not even be aware of the presence of capture device 104) who happens to be captured in video frame 250.

[0134] Dashed shapes 270-272 are shown. Dashed shapes 270-272 can represent object detection by computer vision operations performed by processor 102. Each of dashed shapes 270-272 can include pixel data corresponding to an object detected by the computer vision operation pipeline and / or CNN module 190b. In the illustrated example, dashed shapes 270-272 can be detected in response to a face detection operation performed by the computer vision operation pipeline and / or CNN module 190b. Dashed shapes 270-272 are shown for illustrative purposes. In the example, dashed shapes 270-272 can be a visual representation of object detection (e.g., dashed shapes 270-272 may not appear on the output video frame). In another example, dashed shapes 270-272 can be bounding boxes generated by processor 102 that are displayed on the output video frame to indicate that an object has been detected (e.g., bounding boxes 270-272 can be displayed in a debug mode of the operation).

[0135] Computer vision operations can be configured to detect features of detected objects, behaviors of detected objects, and / or the liveness of detected objects. Object features may include height, length, width, slope, arc length, color, luminous intensity, detected text on the object, movement path, movement speed, movement direction, proximity to other objects, etc. Detected object features may include the object's state (e.g., open, closed, on, off, etc.). Behavior and / or liveness can be determined in response to the object's type and / or the characteristics of the detected object. Although an example video frame 250 is shown, object behavior and / or liveness can be determined by analyzing a sequence of video frames captured over time. For example, movement path and / or movement speed characteristics can be used to determine that an object classified as a person may be walking or running. The types of detected features and / or behaviors can vary depending on the design criteria of a particular implementation.

[0136] In the illustrated example, bounding box 270 may be the face / head of foreground person 252, and bounding box 272 may be the face / head of background person 254. In the example, the settings of processor 102 (e.g., feature set) may define the object of interest as a person and / or face. For example, trees, plants, decorations, furniture, and vehicles may not be objects of interest in the feature set defined for detecting people and / or faces. In the illustrated example, bounding boxes 270 and 272 are shown as having square (or rectangular) shapes. In some embodiments, the shapes of bounding boxes 270-272 corresponding to detected faces may be formed to follow the shape of the face (e.g., an irregular shape following the curve of the face 260-262 of people 252-254).

[0137] Processor 102 and / or CNN module 190b can be configured to implement face detection techniques. Face detection techniques can be performed on RGB images. Processor 102 can cause the face detection techniques to loop (e.g., iteratively perform face detection throughout the example video frame 250) to determine if any faces are within the field of view of lens 160 and / or image sensor 180. In the illustrated example, by iteratively performing the face detection techniques, processor 102 can detect face 260 within bounding box 270 and face 262 within bounding box 272.

[0138] Processor 102 can determine whether any faces are present in the analyzed video frame. If a face has been captured in the video frame, processor 102 can record the size and offset (e.g., position) of the face. In the illustrated example, processor 102 can detect faces 260 and 262. Processor 102 can store the positions of the faces in memory 150.

[0139] In example video frame 250, the dimensions FH1 and FW1 of bounding box 270 are shown. Dimension FH1 may represent the height measurement of face 260 detected by processor 102. Dimension FW1 may represent the width measurement of face 260 detected by processor 102. Similarly, the dimensions FH2 and FW2 of bounding box 272 are shown. Dimension FH2 may represent the height measurement of face 262 detected by processor 102. Dimension FW2 may represent the width measurement of face 262 detected by processor 102. The height and width of each face detected can be stored in memory 150. In this example, memory 150 may store the dimensions FH1 and FW1 of face 260, and the dimensions FH2 and FW2 of face 262.

[0140] Processor 102 can also determine the offsets of faces 260 and 262. This offset can be used to determine the position of each detected face within video frame 250. The offset can be calculated based on the distance from the edge of video frame 250. The offsets of bounding boxes 270-272 can be stored in memory 150.

[0141] In example video frame 250, the horizontal offset LO1 and vertical offset TO1 of bounding box 270 are shown. Horizontal offset LO1 may represent an offset measurement of face 260 measured by processor 102 from the vertical edge of example video frame 250. Vertical offset TO1 may represent an offset measurement of face 260 measured by processor 102 from the horizontal edge of example video frame 250. Similarly, the horizontal offset LO2 and vertical offset TO2 of bounding box 272 are shown. Horizontal offset LO2 may represent an offset measurement of face 262 measured by processor 102 from the vertical edge of example video frame 250 (e.g., the same vertical edge used for offset measurement LO1). Vertical offset TO2 may represent a vertical offset of face 262 measured by processor 102 from the horizontal edge of example video frame 250 (e.g., the same horizontal edge used for offset measurement TO1). The horizontal and vertical offsets of each detected face may be stored in memory 150. In the example, memory 150 may store the horizontal offset LO1 and vertical offset TO1 of face 260, and the horizontal offset LO2 and vertical offset TO2 of face 262.

[0142] In the illustrated example, horizontal offsets LO1-LO2 are shown as measurements relative to the left edge of example video frame 250. In some embodiments, both horizontal offset measurements can be performed from the right edge of example video frame 250. In some embodiments, horizontal offset measurements can be measured from different vertical edges (e.g., the vertical edge closest to the bounding box). In the illustrated example, vertical offsets TO1-TO2 are shown as measurements relative to the top edge of example video frame 250. In some embodiments, both vertical offset measurements can be performed from the bottom edge of example video frame 250. In some embodiments, vertical offset measurements can be measured from different horizontal edges (e.g., the horizontal edge closest to the bounding box). In the illustrated example, two offset measurements are calculated. In some embodiments, offset measurements can be performed at each corner of bounding boxes 270-272. In some embodiments, offset measurements can be performed on each side of bounding boxes that detect irregular shapes (e.g., for each side of a polygon). The type of offset measurement performed can vary depending on the design criteria of a particular implementation.

[0143] Processor 102 can perform various face detection techniques to detect the size and / or position of faces 260-262. In one example, processor 102 can measure the height and width of faces 260-262 by measuring the number of pixels in the height and width of bounding boxes 270-272. In another example, processor 102 can compare the relative number of pixels in bounding boxes 270-272 with objects of known size and / or objects in a reference image. For example, processor 102 can determine the number of pixels per metric for bounding boxes 270 and 272. In the illustrated example, bounding boxes 270-272 are typically shown aligned with the edges of example video frame 250 (e.g., faces 260-262 are directly facing image sensor 180 without rotation). In embodiments where faces 260-262 are rotated relative to example video frame 250, processor 102 can be configured to rotate the bounding boxes to determine the height FH1-FH2 and width FW1-FW2 of the detected faces. In some embodiments, if lens 160 introduces distortion (e.g., barrel distortion), processor 102 may perform dedistortion before detecting faces 260-262. The method for determining the size and / or location of detected faces in a video frame may vary depending on the design criteria of a particular implementation.

[0144] In the illustrated example, bounding box 270 can be significantly larger than bounding box 272. Bounding box 270 can occupy a much larger portion of the area of ​​video frame 250 compared to bounding box 272. In the example, if video frame 250 is a 1920x1080 resolution video frame, bounding box 270 could include a height FH1 of 800 pixels and a width FW1 of 700 pixels (e.g., a 700x800 pixel region of interest), while bounding box 272 could include a height FH2 of 100 pixels and a width FW2 of 100 pixels (e.g., a 100x100 pixel region of interest). Because bounding box 270 includes more pixel data than bounding box 272, processor 102 may need to perform more computations to generate a depth map and / or perform liveness detection for face 260 compared to face 262. However, since face 260 is closer to image sensor 180 and its pixel data may include more detail compared to face 262, processor 102 can reduce video frame 250 while still retaining enough pixel data to generate a depth map and / or perform liveness detection. For example, a smaller face 262 could include enough pixel data to detect liveness. The larger face 260's pixel data can provide additional accuracy but also leads to unnecessary computation. Processor 102 can be configured to determine whether analyzing the larger face 260 involves unnecessary computation and perform reduction to prevent unnecessary computation. For example, reducing face 260 to the size of a smaller face 262 can still produce enough pixel data to detect liveness while reducing the amount of computation used to detect the liveness of face 260.

[0145] refer to Figure 6 , Figure 6 A block diagram illustrating the operation of a processor for adaptive facial depth map generation is shown. An adaptive facial depth map generation component 300 is also shown. The adaptive facial depth map generation component 300 may include various hardware, concept blocks, inputs, and / or outputs that can be used by the device 100 to perform adaptive facial depth generation. The adaptive facial depth map generation component 300 can be represented as a block diagram illustrating the operations performed by the device 100 to reduce the amount of computation performed in generating depth maps from video frames including structured light patterns (SLPs), such as signal DIMAGES, and / or performing post-processing on the generated depth maps.

[0146] The adaptive facial depth map generation component 300 may include a processor 102, a CNN module 190b, video frames 302a-302n, and / or a reference image 304. For illustrative purposes, the processor 102 and the CNN module 190b are shown as separate components. However, the CNN module 190b may be a component implemented by the processor 102. The reference image 304 may be stored in memory 150.

[0147] Processor 102 can be configured to receive a signal VIDEO. The signal VIDEO may include RGB-IR pixel data generated by image sensor 180. The pixel data may include information about the environment and / or objects near device 104 and a structured light pattern SLP projected onto the environment and / or objects. Processor 102 can generate signals (e.g., frames). Signal FRAMES may include video frames 302a-302n. Processor 102 can be configured to process the pixel data, which is arranged into video frames 302a-302n including the structured light pattern SLP. Video frames 302a-302n may be presented to CNN module 190b (e.g., processed internally by processor 102). Processor 102 may be able to use video frames 302a-302n to perform other operations (e.g., generating encoded video frames for display, packaging video frames 302a-302n for transmission using communication module 154, etc.).

[0148] CNN module 190b can be configured to receive video frames 302a-302n, reference image 304, and / or signals (e.g., TOL). CNN module 190b can be configured to generate signals (e.g., LIVEOUT). Signal TOL may include parameters configured to adjust and / or select options for adaptive facial depth generation. Signal LIVEOUT may include output results. CNN module 190b can be configured to receive and / or generate other inputs / outputs. The number, type, and / or format of inputs / outputs transmitted to / from CNN module 190b can vary according to the design criteria of a particular implementation.

[0149] Reference image 304 can be presented from memory 150 to CNN module 190b. Reference image 304 may include video data and / or pixel data of an environment with a structured light pattern. Reference image 304 may include known objects and / or features that can provide a basis for comparison with real-time captured video frames 302a-302n. In an example, reference image 304 may include labels for various objects, features, characteristics, distances, sizes, etc. Reference image 304 can be used to perform block matching with input video frames 302a-302n (e.g., newly captured video data). Reference image 304 can be captured and prepared offline. Reference image 304 can be acquired and preprocessed before the pixel data of input video images 302a-302n is generated by capture device 104 and arranged into video frames by processor 102. In some embodiments, reference image 304 may be manually labeled. Reference image 304 can be used for computer vision operations, 3D reconstruction, depth map generation, etc. Reference image 304 enables 3D reconstruction to be performed when implementing monocular lens 160. Memory 150 can be configured to store multiple reference images 304 of various reference positions and / or objects (e.g., faces) at different distances, providing a basis for comparing structured light patterns (SLPs) captured in real-time video frames 302a-302n. Reference images 304 may include images in IR YUV format. Reference images 304 may include dot patterns associated with various distances to the lens and / or various types of objects. The number of stored reference images 304 and / or various features, characteristics, and / or distances of the structured light patterns (SLPs) can vary according to the design criteria of a particular implementation.

[0150] CNN module 190b may include blocks (or circuits) 310, 312, 314, 316, 318, 320, 322, and / or 324. Circuit 310 may implement an RGB extraction module. Circuit 312 may implement a face detection module. Circuit 314 may include a scaling module. Circuit 316 may implement an IR extraction module. Circuit 318 may implement a cropping module. Circuit 320 may implement a reduction module. Circuit 322 may implement a depth map generation module. Circuit 324 may implement a post-processing module. Circuits 310-324 may be implemented as discrete hardware modules and / or combinations of hardware engines 204a-204n to perform specific tasks. Circuits 310-324 may be conceptual blocks illustrating various techniques implemented by CNN module 190b. CNN module 190b may include other components (not shown). The number, type, and / or arrangement of components in CNN module 190b.

[0151] Video frames 302a-302n can be presented to the RGB extraction module 310 and the IR extraction module 316. The RGB extraction module 310 and the IR extraction module 316 can be configured to extract appropriate content from the video frames 302a-302n for use in subsequent modules in the CNN module 190b.

[0152] The RGB extraction module 310 can receive video frames 302a-302n. The RGB extraction module 310 can be configured to extract RGB data (or RGB video frames) from the RGB-IR video frames 302a-302n. The RGB extraction module 310 can be configured to generate a signal (e.g., RGB). The RGB signal may include RGB channel data extracted from the video frames 302a-302n. The RGB signal may include RGB information without a structured light pattern (SLP). The RGB signal can be presented to the face detection module 312.

[0153] The face detection module 312 can be configured to receive an RGB signal. The RGB signal can include RGB channel information from video frames 302a-302n. The face detection module 312 can perform various face detection techniques to detect faces 260-262. Face detection techniques can be performed on the RGB information from video frames 302a-302n. The face detection module 312 can be configured to generate a signal (e.g., FDATA). The FDATA signal can include information about faces detected in video frames 302a-302n. In an example, the FDATA signal can include face heights FH1-FH2, face widths FW1-FW2, and / or offset information LO1-LO2 and TO1-TO2 about faces 260-262. The FDATA signal can be presented to a scaling module 314 and / or a cropping module 318. This can be combined with... Figure 5 This describes the details of the face detection performed by the face detection module 312.

[0154] The scaling ratio module 314 can be configured to receive signals FDATA, TOL, and / or a signal (e.g., EXP). The scaling ratio module 314 can be configured to determine a scaling ratio for the reduction operation. The scaling ratio module 314 can be configured to generate a signal (e.g., R). The signal R may include the scaling ratio calculated in response to the signals FDATA, TOL, and / or EXP. The scaling ratio signal R can be presented to the reduction module 320.

[0155] The signal EXP may include post-processing parameters. In the illustrated example, the post-processing parameters may include expected depth pixel values ​​for liveness detection performed by the post-processing module 324. In some embodiments, for other types of post-processing operations, the post-processing parameters may include additional data regarding sufficient pixel data.

[0156] The scaling ratio module 314 can be configured to capture a desired number of depth pixels. Liveness detection techniques can have a range of desired numbers of facial depth pixels. If the facial depth resolution exceeds a minimum threshold for the number of facial depth pixels, the liveness detection accuracy may improve slightly. Generally, as the facial depth resolution increases, the computation time for liveness detection performed by the post-processing module 324 may increase linearly, but the liveness detection accuracy may hardly improve. The tolerance parameter TOL can be set by the user to adjust the desired depth pixels depending on the choice between detection accuracy and computation speed.

[0157] The scaling ratio module 314 can be configured to calculate the scaling ratio R of the structured light pattern image given the size of the face region (e.g., signal FDATA) and the expected depth pixels (e.g., signal EXP). The scaling ratio module 314 can calculate the scaling ratio R based on formula [1]:

[0158] [1]R=(E / W x H)+T

[0159] In Equation [1], E can represent the expected depth pixel (e.g., from signal EXP), W can represent the face width (e.g., from signal FDATA FW1), H can represent the face height (e.g., from signal FDATA FH1), R can represent the scaling ratio, and signal T can represent the tolerance parameter (e.g., signal TOL).

[0160] Different sizes of facial regions may result in different scaling ratios. The larger the size of the facial region, the smaller the result calculated for that scaling ratio. In one example, if a liveness detection technique requires 10,000 depth pixels to detect fraud (e.g., distinguishing between fake and real faces), with a tolerance of 0.1 and a facial region of 100x200 pixels, the scaling ratio could be 0.6.

[0161] The IR extraction module 316 can receive video frames 302a-302n. The IR extraction module 316 can be configured to extract IR data (or IR video frames) from the RGB-IR video frames 302a-302n. The IR extraction module 316 can be configured to generate a signal (e.g., IR). The IR signal may include IR channel data extracted from the video frames 302a-302n. The IR signal may include a structured light pattern SLP. The IR signal may include a full-resolution IR image. The IR signal can be presented to the reduction module 320.

[0162] The cropping module 318 can be configured to perform a cropping operation on a full-resolution structured light image from the signal IR. The cropping module 318 can remove and / or delete multiple pixels from the full-resolution structured light image. Removing and / or deleting pixels can reduce the size of the full-resolution structured light image and / or change its aspect ratio.

[0163] The cropping module 318 can perform a cropping operation to remove and / or delete pixels in response to the signal FDATA. The signal FDATA may include offset values ​​(e.g., LO1, TO1, LO2, TO2, etc.) and dimensions (e.g., FW1, FH1, FW2, FH2, etc.) that can be used to locate the face in the full-resolution structured light image. For example, in response to the offset values ​​and / or dimension values ​​in the signal FDATA, the cropping module 318 can determine the position of the bounding box 270-272 of face 260-262 in a video frame. The cropping operation can be performed on each face detected in the full-resolution structured light image in the signal IR.

[0164] A cropping operation can be performed to remove pixel values ​​that do not correspond to the face position. For example, after the cropping module 318 performs the cropping operation, most of the remaining pixel values ​​can correspond to the face position in the original video frames 302a-302n. In one example, the cropped video frame can be rectangular. Depending on the detected face position and orientation, each cropped video frame can be a rectangle with different dimensions (e.g., different aspect ratios).

[0165] The cropping module 318 can generate a signal (e.g., a CIR). The signal CIR can include a cropped full-resolution structured light image. For example, the IR image in the signal CIR can have a smaller size (e.g., fewer pixel values) compared to the IR image in the signal IR. The signal CIR can be presented to the reduction module 320.

[0166] The reduction module 320 can be configured to reduce a cropped structured light image from the signal CIR. The reduction module 320 can reduce the cropped structured light image derived from the extracted IR data and then cropped from real-time video frames 302a-302n and / or the structured light reference image 304. After the scaling ratio module 314 calculates the scaling ratio R, the reduction module 320 can perform a reduction operation on the cropped full-resolution real-time IR image extracted from video frames 302a-302n and the reference image 304 to generate a structured light image reduced to the desired size. The desired size can retain sufficient pixel data for successful post-processing while eliminating unnecessary calculations used to generate the depth map.

[0167] The reduction module 320 can generate a signal (e.g., DLIMG) and a signal (e.g., DRIMG). The DLIMG signal can include a reduced real-time structured light image. The reduced real-time structured light image DLIMG can be generated in response to the real-time structured light IR image extracted from video frames 302a-302n and a scaling ratio R. The DRIMG signal can include a reduced reference structured light image. The reduced reference structured light image DRIMG can be generated in response to a reference structured light video frame 304 and a scaling ratio R. In one example, the reduced reference structured light image DRIMG can be generated in response to a reduction operation performed on a single reference image. In another example, the reduced structured light reference image DRIMG can be generated in response to multiple reference images 304. In some embodiments, the scaling ratio R can be determined in real time for a single reference image 304. For example, the reduced real-time structured light image DLIMG and the reduced reference structured light image DRIMG can be generated in real time after the scaling ratio R is calculated. In some embodiments, the scaling ratio can be pre-calculated using multiple reference images 304. The reduced real-time structured light image DLIMG and the reduced reference structured light image DRIMG can be presented to the depth map module 322.

[0168] Generating reduced real-time structured light images (DLIMG) and reduced reference structured light images (DRIMG) enables the depth map module 322 to generate depth maps more efficiently than generating depth maps from high-resolution real-time images 302a-302n and reference image 304. For example, depth map generation can be accelerated because less computation is required on the reduced real-time structured light images (DLIMG) and reduced reference structured light images (DRIMG) compared to the computation required on the full-resolution structured light images (IR, or cropped full-resolution structured light images) and the full-resolution reference image 304, thus requiring less computation time.

[0169] Compared to not performing a reduction operation, the reduction operation performed by reduction module 320 can utilize additional resources (e.g., increased computational load may be required to perform the reduction operation to generate a reduced real-time structured light image DLIMG and a reduced reference structured light image DRIMG). The computational complexity of performing the reduction operation may result in fewer computations and / or less computation time compared to the additional computation required to generate a depth map using full-resolution IR images extracted from full-resolution real-time video frames 302a-302n and reference image 304. In one example, the time cost of performing the reduction operation by the video processing hardware of reduction module 320 for a 1920x1080 image input could be approximately one millisecond. For example, the additional time cost of the reduction operation is negligible compared to the time saved in generating a depth map from the reduced real-time structured light image DLIMG and the reduced reference structured light image DRIMG instead of from the full-resolution IR images extracted from real-time video frames 302a-302n and reference image 304. The reduction in computations due to the smaller size of the scaled real-time structured light image DLIMG and the scaled reference structured light image DRIMG is linearly proportional to the size change of the full-resolution IR image.

[0170] Depth map module 322 can be configured to generate depth maps. Depth map module 322 can be configured to receive a reduced real-time structured light image (DLIMG) and a reduced reference structured light image (DRIMG) from reduction module 320. Depth map module 322 can be configured to generate signals (DIMAGES). Signals (DIMAGES) can be generated in response to the reduced real-time structured light image (DLIMG) and the reduced reference structured light image (DRIMG). In an example, depth map module 322 can be configured to compare the reduced real-time structured light image (DLIMG) with the reduced reference real-time structured light image (DRIMG) to find matching patterns. Depth maps (DIMAGES) can be presented to post-processing module 324. Details of depth map generation can be described in connection with U.S. Patent Application No. 17 / 479,034, filed September 20, 2021, the appropriate portions of which are incorporated herein by reference.

[0171] When depth map module 322 generates depth maps DIMAGES based on a reduced real-time structured light image DLIMG and a reduced reference structured light image DRIMG, it can generate depth map images DIMAGES faster than generating depth maps using video frames 302a-302n and reference image 304. Since the reduced real-time structured light image DLIMG and the reduced reference structured light image DRIMG can include consistent resolution values, depth maps DIMAGES with consistent resolution can be generated. The depth maps DIMAGES generated in response to the reduction operation can include consistent depth maps.

[0172] Performing the reduction operation using a scaling ratio R ensures that a sufficient number of depth pixels are retained for both the reduced real-time structured light image (DLIMG) and the reduced reference structured light image (DRIMG). Retaining a sufficient number of depth pixels enables the generation of depth maps (DIMAGES) while avoiding excessive computation of depth information. When using the reduced real-time structured light image (DLIMG) and the reduced reference structured light image (DRIMG), retaining a sufficient number of depth pixels prevents block matching failures. Retaining a sufficient number of depth pixels ensures that the depth maps (DIMAGES) include the expected range of depth pixels for liveness detection performed by the post-processing module 324.

[0173] Implementing the cropping module 318 before the reduction module 320 allows cropping of the real-time IR image to be performed before the reduction operation. Performing cropping before reduction can reduce the operation time of the reduction operation performed by the reduction module 320 (e.g., performing reduction on a cropped image with fewer pixel values). The cropping operation can be performed on the real-time IR image in the signal CIR. The cropping operation may not be performed on the reference image 304. For example, the signal DLIMG may include the reduced and cropped real-time image, and the signal DRIMG may include the reduced reference image. The signal DLIMG may include the facial field of view, while the signal DRIMG may include the entire field of view. The depth map module 322 can analyze the cropped and reduced structured light pattern in the signal DLIMG and search for matching patterns between the reduced reference structured light images in the signal DRIMG.

[0174] A scaling ratio R can be calculated to dynamically create depth maps DIMAGES within a resolution range corresponding to the size of the facial region detected in the RGB channels of real-time video frames 302a-302n. The size of the facial region can be a variable size in each video frame 302a-302n, partially based on the distance between the person 252-254 and the image sensor 180. The tolerance value TOL can be a configurable value that can be set by the user (or pre-configured for device 100) to allow the user to adjust the scaling operation between detection accuracy and computation speed. A range of values ​​for the tolerance value TOL can be selected to ensure that the scaling ratio R results in a sufficient number of depth pixels.

[0175] Post-processing module 324 can be configured to perform one or more post-processing techniques. In the illustrated example, the post-processing technique may include liveness detection. Typically, the liveness detection technique implemented by post-processing module 324 may use a CNN. However, apparatus 100 can be configured to receive a signal EXP to generate a scaling ratio R, even if the liveness detection technique is not implemented using a CNN (e.g., post-processing module 324 is implemented without using CNN module 190b). The type of post-processing performed may vary depending on the design criteria of a particular implementation.

[0176] The post-processing module 324 can be configured to determine whether a detected face is a real face or a fake face (e.g., detecting fraud) in response to a depth map generated by the depth map module 322. The post-processing module 324 can be configured to receive the signal DIMAGES. The post-processing module 324 can be configured to generate the signal EXP and the signal LIVEOUT. The signal EXP can be the expected number of depth pixels for the post-processing module 324 to perform liveness detection techniques.

[0177] A signal LIVEOUT can be generated in response to depth maps DIMAGES. For example, the signal LIVEOUT may include a determination (e.g., Boolean value, probability percentage, confidence value, etc.) of whether a face detected in video frames 302a-302n is real (e.g., live) or fake. Processor 102 can use the signal LIVEOUT for various functions. In an example, processor 102 may generate a signal CTRL in response to the signal LIVEOUT (e.g., to verify that an authorized person has been detected to allow access to a secure area). The use of the results of the signal LIVEOUT (e.g., the results of a liveness detection performed by post-processing module 324) may vary depending on the design criteria of a particular implementation.

[0178] Compared to generating depth maps using full-resolution video frames 302a-302n and reference image 304 without downscaling, performing downscaling operations in response to downscaled real-time structured light images (DLIMG) and downscaled reference structured light images (DRIMG) to generate depth maps (DIMAGES) reduces facial depth computation time. Furthermore, facial depth computations performed by depth map module 322 can be performed consistently. Providing consistent depth maps (DIMAGES) to post-processing module 324 can offer advantages for post-processing (e.g., liveness detection). In one example, providing consistent facial depth input to post-processing module 324 allows liveness detection techniques to be trained on a specific number of pixels, rather than across all resolutions. In another example, providing consistent facial depth input to post-processing module 324 allows liveness detection techniques to detect facial liveness at a specific number of pixels. Consistent facial depth input reduces the complexity of post-processing operations.

[0179] The signal DIMAGES can include depth maps with a consistent number of total pixels (e.g., consistent facial depth density) for the reduced facial region. For example, reduced and cropped structured light images can have different aspect ratios but can result in depth maps with a consistent facial depth density. In one example, after cropping and resizing, one of the real-time structured light images can have a 100x100 pixel facial region, while another real-time structured light image can have a consistent depth density with a 125x80 pixel facial region (e.g., different aspect ratios, but each with 10,000 pixels). When the scaling ratio R is calculated in real time (e.g., using a single reference image), the total number of pixels in each depth map can be approximately the same after reduction. When the scaling ratio R is pre-calculated (e.g., using multiple reference images), the final reduction ratio can be less than the scaling ratio calculated in real time in order to keep the density consistently higher than the target density.

[0180] refer to Figure 7 , Figure 7 A diagram illustrating the extraction of IR video frames for reduction is shown. An example RGB / IR dispatch 350 is shown. Example RGB / IR dispatch 350 illustrates the extraction of RGB and IR video frames from video frames 302a-302n generated by processor 102.

[0181] Example RGB / IR dispatch 350 may include processor 102, RGB extraction module 310, IR extraction module 316, block (or circuitry) 352, and / or block (or circuitry) 354. Block 352 may represent an IR image channel. Block 354 may represent an RGB image channel. Although video frames 302a-302n are shown as being within processor 102, each of the RGB extraction module 310, IR extraction module 316, IR image channel 352, and / or RGB image channel 354 may be implemented within processor 102 and / or CNN module 190b.

[0182] In the illustrated example, video frames 302a-302n can be generated by generating a structured light pattern SLP in one-third of the input frames using a structured light projector 106. As an illustrative example of the structured light pattern SLP, a dot 360 is shown in video frames 302a and 302d (e.g., every third video frame). Although the dot 360 is shown with the same pattern for illustrative purposes, the dot pattern can be different for each video frame in video frames 302a-302n that capture the structured light pattern SLP. The dot pattern 360 can be captured every third video frame in video frames 302a-302n by timing the projection of the structured light pattern SLP onto one-third of the input frames. Although the timing of the one-third structured light pattern projection is shown, the apparatus 100 can be implemented with other rules to dispatch and / or extract IR and RGB images from video frames 302a-302n.

[0183] Video frames 302a-302n can be presented to the RGB extraction module 310 and the IR extraction module 316. The RGB extraction module 310 can generate an RGB signal. The RGB signal can be transmitted through RGB channel 354 (e.g., a subset of the RGB values ​​of video frames 302a-302n). The IR extraction module 316 can generate an IR signal. The IR signal can be transmitted through IR channel 352 (e.g., a subset of the IR values ​​of video frames 302a-302n).

[0184] IR channel 352 may include IR images 370a-370k. IR images 370a-370k may each include a dot pattern 360. IR images 370a-370k may each include a full-resolution IR image. IR image 370a may correspond to video frame 302a (e.g., having the same dot pattern 360), and IR image 370b may correspond to video frame 302d. Since the structured light pattern SLP is projected and captured for a subset of video frames 302a-302n, IR channel 352 may include fewer IR images 370a-370k than the total number of video frames 302a-302n generated. In the illustrated example, IR channel 352 may include each one-third of video frames 302a-302n.

[0185] The structured light pattern SLP can be displayed on the IR channel data. The IR data can be extracted from the output of the RGB-IR sensor 180 by the IR extraction module 316. The IR images 370a-370k can be formatted as IR YUV images. The IR images 370a-370k, including the dot pattern 360 in IR YUV image format, can be presented to the cropping module 318. The cropping module 318 can crop the IR images 370a-370k in IR YUV format, including the structured light dot pattern 360. The cropped structured light image can be presented to the reduction module 320. The reduction module 320 can reduce the cropped IR image (e.g., signal CIR) in IR YUV format, including the structured light dot pattern 360.

[0186] RGB channel 354 may include RGB images 372a-372m. RGB images 372a-372m may not include dot pattern 360. For example, structured light projector 106 may be turned off periodically during the capture of RGB images 372a-372m. RGB image 372a may correspond to video frame 302b (e.g., having the same RGB pixel data), RGB image 372b may correspond to video frame 302c, RGB image 372c may correspond to video frame 302e, RGB image 372d may correspond to video frame 302f, and so on. Since the structured light pattern SLP is projected and captured for a subset of video frames 302a-302n (e.g., IR images 370a-370k) and turned off for another subset of video frames 302a-302n (e.g., RGB images 372a-372a-372m), the RGB channel 354 can include fewer RGB images 372a-372m than the total number of generated video frames 302a-302n but more than the total number of IR images 370a-370k.

[0187] RGB data can be extracted from the output of the RGB-IR sensor 180 by the RGB extraction module 310. RGB images 372a-372m can be formatted as RGB image formats. RGB images 372a-372m can be transmitted to the face detection module 312. The reduction factor (e.g., scaling ratio R) can depend on the CNN-based face detection results determined from the RGB images 372a-372m (e.g., the bounding box resolution of the detected faces, the offset position of the faces, etc.). Face detection performed by the face detection module 312 can be performed on RGB data (e.g., without dot pattern 360). To generate an adaptive face depth map, the RGB images 372a-372m may not need to be cropped or reduced (e.g., the RGB signals may not be presented to the cropping module 318 and / or reduction module 320). However, for other video processing operations performed by the processor 102, cropping and / or reduction can be performed on the RGB images 372a-372m.

[0188] refer to Figure 8 , Figure 8 A diagram illustrating the reduction of a structured light pattern is shown. A reduction example 400 is shown. The reduction example may include one of a video frame 302i (e.g., including RGB and IR data), a full-resolution IR image 370i (e.g., including a structured light pattern SLP), a cropped IR image 402i, and a reduced IR image 404.

[0189] Video frame 302i may be a representative example of one of video frames 302a-302n. Face 260 is shown. Video frame 302i may illustrate how dot pattern 360 is projected onto face 260 and other portions of video frame 302i. The structured light pattern SLP may include a maximum dot 410, a large dot 412, a small dot 414, and / or a minimum dot 416. Details of generating a structured light pattern SLP and / or analyzing dot patterns and / or object distances based on a structured light pattern SLP can be described in U.S. Application No. 17 / 479,034, filed September 20, 2021, the appropriate portions of which are incorporated herein by reference.

[0190] The largest dot 410 can be located in video frame 302i closest to the camera 160. The largest dot 410 can be clustered on the nose of the face 260. The large dot 412 can be located in video frame 302i close to the camera 160, but not as close as the largest dot 410. The large dot 410 can be located on the face 260. The small dot 414 can be located in video frame 302i further away from the camera 160 than dot 412. The small dot 414 can be located on the ears, neck, and shoulders of the person in video frame 302i. The smallest dot 416 can be located in video frame 302i furthest from the camera 160. The smallest dot 416 can be located around the face 260 (e.g., on the background behind the person in video frame 302i). Although four sizes of points 410-416 are shown for illustrative purposes, the number of point sizes and / or the size of each point in the structured light pattern may vary depending on the position and / or characteristics of various objects in video frames 302a-302n.

[0191] A full-resolution IR image 370i can be generated in response to an operation of the IR extraction module 316 on the video frame 302i. The full-resolution IR image 370i can have the same dimensions (e.g., resolution, dimensions, aspect ratio, etc.) as the video frame 302i. The full-resolution IR image 370i can have the same scaling as the video frame 302i. The full-resolution IR image 370i can depict points 410-416 that do not display RGB data for the face 260. Points 410-416 in the full-resolution IR image 370i are shown as having the same pattern and dimensions as points 410-416 in the video frame 302i.

[0192] The cropped IR image 402i can be a representative example of cropped IR images 402a-402k in the signal CIR. The cropped IR image 402i can be generated in response to operation of the full-resolution IR image 370i by the cropping module 318. The cropped IR image 402i can be smaller than the full-resolution IR image 370i (e.g., fewer pixels, lower resolution, smaller dimensions, different aspect ratio, etc.). The cropped IR image 402i can have the same scaling as the full-resolution IR image 370i. The cropped IR image 402i can show points 410-416 without RGB data, and some pixel data from the full-resolution IR image 370i is removed and / or deleted. Points 410-416 in the cropped IR image 402i that are not deleted and / or removed are shown as having the same pattern and size as points 410-416 in the full-resolution IR image 370i.

[0193] The cropping module 318 can use the facial position and / or size information in the signal FDATA to remove and / or delete pixel data that does not correspond to the face 260 from the full-resolution IR image 370i. In the example shown, some points 414 corresponding to the neck of the face 260 and some points 416 corresponding to the background surrounding the face 260 have been deleted. For example, the size of the face represented by points 410-414 in the cropped IR image 402i can be the same as the size of the face represented by points 410-414 in the full-resolution IR image 370i (e.g., points 410-416 may have the same scaling). Because many pixels that do not correspond to the position of the face 260 have been deleted, the cropped IR image 402i can be smaller (e.g., in terms of the total number of pixels). For example, the cropped IR image 402i can have fewer total pixels than the full-resolution IR image 370i, but the number of pixels representing the face 260 can be the same in both the cropped IR image 402i and the full-resolution IR image 370i.

[0194] The reduced IR image 404 can be generated by the reduction module 320 in response to the cropped IR image 402i and the scaling ratio R. In some embodiments, when cropping is not performed, the reduced IR image 404 can be generated by the reduction module 320 in response to the full-resolution IR image 370i and the scaling ratio R. The target resolution (or target pixel density) of the reduced IR image can be predefined by post-processing techniques performed after the generation of depth maps DIMAGES. The scaling ratio R used by the reduction module 320 can be varied to accommodate changes in facial resolution in the corresponding RGB images 372a-372m (e.g., facial movement closer to or further from lens 160).

[0195] The generation of the reduced IR image 404 can reduce the IR data. Reduced IR data can lead to the loss and / or alteration of some point patterns in the captured structured light pattern SLP. Determining the scaling ratio R can prevent data loss from the structured light pattern SLP from affecting the generation of depth maps DIMAGES and / or the results of post-processing. For example, the scaling ratio R can ensure that a sufficient amount of IR data is retained to ensure the generation of depth maps DIMAGES for implementing post-processing techniques (e.g., liveness detection). In one example, the reduction performed by the reduction module 320 may include a bilinear technique. In another example, the reduction performed by the reduction module 320 may include a bicubic technique. Other reduction techniques (e.g., Lanczos, nearest neighbor, etc.) can be implemented. The type of reduction technique performed can vary depending on the design criteria of a particular implementation.

[0196] The reduced IR image 404 may include a maximum reduced point 420, a large reduced point 422, a small reduced point 424, and / or a minimum reduced point 426. The maximum reduced point 420 typically corresponds to the position and / or pattern of the maximum point 410 in the cropped IR image 402i. The large reduced point 422 typically corresponds to the position and / or pattern of the large point 412 in the cropped IR image 402i. The small reduced point 424 typically corresponds to the position and / or pattern of the small point 414 in the cropped IR image 402i. The minimum reduced point 426 typically corresponds to the position and / or pattern of the minimum point 416 in the cropped IR image 402i. During the reduction process, some IR data may be lost and / or altered. For example, the positions and / or patterns of the reduced points 420-426 may not perfectly match the positions and / or patterns of points 410-416.

[0197] The reduced IR image 404 may include missing data 428 and / or merged points 430. In the illustrated example, missing data 428 may correspond to the location of one of the smallest points 416 in the cropped IR image 402i. Missing data 428 may represent a location where points no longer exist from the structured light pattern SLP after reduction. In the illustrated example, merged data 430 may be a larger point at a location corresponding to more than one smallest point 416 in the cropped IR image 402i (or the original full-resolution IR image 370i). Merged data 430 may represent a location where multiple data points (e.g., points) that may be close together have been misinterpreted as larger points. Typically, missing data and / or merged data may correspond to the smallest point. However, point patterns at different distances from lens 160 may be misinterpreted during the reduction process (e.g., one of the largest points 410 may be misinterpreted as a smaller point, more than one large point 412 may be merged together as the largest point, etc.).

[0198] The missing data 428 and / or merged data 430 may represent IR data that was misinterpreted during the downscaling process, which could lead to errors in the generation of depth map DIMAGES. Determining the scaling ratio R allows the device 100 to limit the amount of data loss. Data loss can be limited so that the computational cost of generating depth map DIMAGES is reduced compared to generating depth maps directly from cropped IR images 402a-402k (or full-size IR images 370a-370k), while still providing enough data to enable post-processing techniques.

[0199] The image used for reduction can be a cropped IR image 402a-402k including the exposed structured light point pattern (or a full-resolution IR image 370a-370k if no cropping is performed). After the reduction operation, the detail of the structured light pattern may be reduced with the dimensions and / or scaling of the image. The loss of detail from reduction may include the total number of pixels, the size and / or pattern of the captured points, and the accuracy of the feature points.

[0200] The reduction in the total number of pixels in the downsized IR image 404 may not be a disadvantage, as the loss of total pixels can result in faster computation. To provide a smaller number of points in the structured light pattern, a loss in the total number of pixels can be desirable. Fewer points in the downsized IR image 404 can result in faster depth map generation compared to using the cropped IR image 402i (or the full-resolution IR image 370i). When there is excessive depth map information (e.g., more structured light data points than are required for successful post-processing), a scaling ratio R can be calculated to ensure that the required (e.g., sufficient) number of depth resolutions is calculated to provide the target facial depth density by controlling the scaling factor and / or location of the facial region.

[0201] The loss of size and / or pattern of the captured points after downscaling may increase the difficulty of performing matching techniques on the point pattern between real-time video frames 302a-302n and reference image 304. After downscaling, some points of the structured light pattern SLP may become blurred and / or merge into one. Instead of performing a direct pixel comparison between the downscaled real-time image 404 and reference image 304, alternative comparison techniques can be implemented. In the example, block matching techniques can be implemented to generate depth maps DIMAGES from the downscaled images (e.g., both the real-time cropped IR images 402a-402k and reference image 304 can be downscaled).

[0202] The decrease in feature point accuracy may be due to a reduction in the resolution of the structured light pattern. The density of the depth information calculated by the depth map generation module 322 can be reduced by the same scaling ratio. The scaling process can be performed on the nearest object (e.g., a nearby face 260 shown in video frame 250 instead of a background face 262). The closer the object is to the lens 160, the greater the difference and density of the depth information that can be generated for the object. The greater the difference and density of the depth information for nearby objects, the more likely the loss of accuracy will be offset. For example, compensation can result in accuracy comparable to that of small faces (e.g., distant faces) in the original real-time image. The tolerance parameter TOL allows the user to adjust whether to generate depth maps with greater accuracy or faster computation speed.

[0203] refer to Figure 9 ,Figure 9 A diagram illustrating the reduction of IR video frames is shown. Data processing 450 of IR images 370a-370k is shown. Data processing 450 can illustrate how processor 102 processes regions of interest of different sizes from facial data. Data processing 450 may include full-resolution IR images 370a-370k, cropped IR images 402a-402k, reduced IR images 404a-404k, and post-processing module 324.

[0204] Full-resolution IR images 370a-370k may include IR images with a consistent resolution and / or aspect ratio (e.g., the resolution and / or aspect ratio of the RGB-IR sensor 180). In the illustrated example, full-resolution IR images 370a-370k may include IR images with rectangular resolution in the lateral direction (e.g., 3840x2160, 1920x1080, 640x480, etc.). Full-resolution IR image 370a may include structured light spots 452a-452m corresponding to a face (other spots may be captured, but are not shown for illustrative purposes). In the illustrated example, structured light spots 452a-452m are primarily shown on the left side of full-resolution IR image 370a. Full-resolution IR image 370b may include structured light spots 454a-454m corresponding to a face (other spots may be captured, but are not shown for illustrative purposes). In the illustrated example, structured light spots 454a-454m are primarily shown in the center of full-resolution IR image 370b. The full-resolution IR image 370k may include structured light dots 456a-456m corresponding to a face (other dots may be captured, but are not shown for illustrative purposes). In the illustrated example, structured light dots 456a-456m are primarily shown on the lower right side of the full-resolution IR image 370k. The size, resolution, and / or dot pattern of the IR images 370a-370k may vary according to the design criteria of a particular implementation.

[0205] The cropping operation can be performed by the cropping module 318. The cropped IR images 402a-402k can be generated by the cropping module 318 based on facial data FDATA in response to the full-resolution IR images 370a-370k. The cropped IR images 402a-402k can be illustrated as corresponding to the full-resolution IR images 370a-370k, respectively.

[0206] The cropped IR images 402a-402k may include IR images with different aspect ratios, resolutions, and dot patterns. In the illustrated example, the cropped IR image 402a may include an IR image with a rectangular resolution (e.g., 3600x2160, 1400x1080, 500x480, etc.) in the lateral direction. The cropped IR image 402a may include structured light dots 452a-452m from the full-resolution IR image 370a, which correspond to the location of a face, while other pixel data is removed and / or eliminated. As an illustrative example, the structured light dots 452a-452m are shown as a regular pattern of small dots. In the illustrated example, the cropped IR image 402b may include an IR image with a rectangular shape (e.g., 1000x2000, 700x1000, 400x800, etc.) in the longitudinal direction. The cropped IR image 402b may include structured light spots 454a-454m from the full-resolution IR image 370b, which correspond to the location of a face, while other pixel data is removed and / or eliminated. As an illustrative example, the structured light spots 454a-454m are shown as an irregular pattern of large dots. In the illustrated example, the cropped IR image 402k may include an IR image with a square shape (e.g., 2000x2000, 1000x1000, 800x800, etc.). The cropped IR image 402k may include structured light spots 456a-456m from the full-resolution IR image 370k, which correspond to the location of a face, while other pixel data is removed and / or eliminated. As an illustrative example, structured light spots 456a-456m are shown as a regular pattern of small dots.

[0207] The size of the facial region in each of the full-resolution IR images 370a-370k can be a variable size based on the distance between the person and the image sensor 180 that captures pixel data used to generate video frames 302a-302n. The variable size of the facial region can result in each cropped IR image 402a-402k having a different aspect ratio, resolution, and / or dot pattern.

[0208] The reduction operation can be performed by the reduction module 320. The reduced IR images 404a-404k can be generated by the reduction module 320 in response to the cropped IR images 402a-402 based on the scaling ratio R. The reduced IR images 404a-404k shown can correspond to the cropped IR images 402a-402k (and the full-resolution IR images 370a-370k), respectively.

[0209] After the reduction operation, the output reduced IR images 404a-404k may have lower resolution and scaling compared to the input IR images 370a-370k. However, the aspect ratio of the output reduced IR images 404a-404k may be similar to (or slightly different from) the cropped IR images 402a-402k. The reduction operation can maintain the same aspect ratio of the detected faces as the original aspect ratio of the faces in the cropped IR images 402a-402k (e.g., measured from facial data FDATA).

[0210] After the downscaling process, the aspect ratio of the downscaled IR images 404a-404k can be the same as that of the cropped IR images 402a-402k, but with reduced resolution. For example, the shape of each scaled IR image 404a-404k can be different, corresponding to different facial shapes (e.g., aspect ratio) in the cropped IR images 402a-402k. In the illustrated example, the downscaled IR image 404a can have the same aspect ratio as the input cropped IR image 402a, with rectangular resolution in the lateral direction but with reduced resolution (e.g., from 3600x2160 to 1400x1080, from 1400x1080 to 500x480, from 500x480 to 300x240, etc.). In the illustrated example, the reduced IR image 404b can have the same aspect ratio as the input cropped IR image 402b, but with a rectangular resolution in the vertical direction, though the resolution is reduced (e.g., from 1000x2000 to 800x1600, from 700x1000 to 350x500, from 400x800 to 100x200, etc.). In the illustrated example, the reduced IR image 404k can have the same aspect ratio as the input cropped IR image 402k, but with a square resolution, though the resolution is reduced (e.g., from 2000x2000 to 1500x1500, from 1000x1000 to 500x500, from 800x800 to 100x100, etc.). The amount of reduction performed can vary depending on the size of the facial region, the number of depth pixels used for post-processing operations, etc.

[0211] The reduction operation can reduce the size and / or number of dots in a structured light pattern. In the illustrated example, the reduced IR image 404a may include reduced dot patterns 462a-462k, the reduced IR image 404b may include reduced dot patterns 464a-464k, and the reduced IR image 404k may include reduced dot patterns 466a-466k. Structured light dots 452a-452m may be reduced to reduced structured light dots 462a-462k, structured light dots 454a-454m may be reduced to reduced structured light dots 464a-464k, and structured light dots 456a-456m may be reduced to reduced structured light dots 466a-466k (e.g., fewer reduced dots compared to the original IR image). After reduction, some small structured light dots may disappear or may merge with other dots (e.g., data loss). The reduced IR images 404a-404k can be presented to the post-processing module 324 as input for post-processing techniques (e.g., liveness detection).

[0212] The entire IR image 370a-370n can be cropped and then reduced. The IR image can be cropped down to just the facial region for liveness detection. Even with data loss in the reduced structured light spots, liveness detection can be unaffected due to the calculated scaling ratio R. By applying cropping and then an adaptive scaling ratio for reduction, each facial region can potentially achieve a similar target depth resolution (e.g., similar total number of pixels but not identical dimensions, as each face may still have a slightly different shape and aspect ratio than the original image). The target facial depth resolution can be similar after reduction. The resolution of the original IR image 370a-370n and the structured light spot pattern is unrestricted. (As related to...) Figure 5 As shown, the cropping to be performed can be determined based on offset values ​​(e.g., LO1 and TO1) and facial dimensions (e.g., FW1 and FH1).

[0213] refer to Figure 10 The diagram illustrates method (or process) 500. Method 500 can generate an adaptive facial depth map. Method 500 generally includes steps (or states) 502, 504, 506, 508, a decision step (or state) 510, 512, 514, 516, 518, and 520.

[0214] Step 502 can initiate method 500. In step 504, processor 102 can receive pixel data (e.g., signal VIDEO) having a structured light pattern SLP. For example, capture device 104 can generate the pixel data, and structured light projector 106 can activate and deactivate the structured light pattern SLP according to a predetermined timing of signal SL_TRIG. For example, video frames 302a-302n may include pixel data corresponding to when structured light source 186 is activated and other pixel data corresponding to when structured light source 186 is deactivated. Next, in step 506, processor 102 can process the pixel data, which is arranged into video frames. In step 508, processor 102 can perform computer vision operations on video frames 302a-302n. Next, method 500 can move to decision step 510.

[0215] In decision step 510, CNN module 190b can determine whether a face is captured in video frames 302a-302n. In the example, RGB extraction module 310 can extract RGB channels 354 from video frames 302a-302n to enable face detection module 312 to perform face detection. If no face is detected in the video frames, then method 500 can return to step 504. If a face is detected in the video frames, then method 500 can move to step 510. In step 510, face detection module 312 can detect the size of face region 270. Face detection module 312 can also detect the offset of face region 270 in the video frames. Next, method 500 can move to step 514.

[0216] In step 514, the scaling ratio module 314 can determine the scaling ratio R based on the size of the facial region 270 (e.g., dimensions FH1 and FW1), the tolerance value TOL, and the number of depth pixels EXP used for post-processing techniques. For example, the post-processing module 324 can provide a signal EXP indicating the number of depth pixels. Next, in step 516, the IR extraction module 316 can extract the structured light pattern (e.g., dot pattern 360) and / or IR images 370a-370k from video frames 302a-302n. The cropping module 318 can crop the full-resolution IR images 370a-370k into cropped IR images 402a-402k based on the position and / or dimensions of the face in the signal FDATA. Next, method 500 can move to step 518.

[0217] In step 518, the reduction module 320 can generate a reduced IR image DLIMG (e.g., in response to the CIR signal) and a reduced reference image DRIMG (e.g., in response to reference image 304) based on a scaling ratio R. Next, in step 520, the depth map module 322 can generate depth maps DIMAGES in response to the reduced IR image DLIMG and the reduced reference image DRIMG. The depth maps DIMAGES can be generated with a consistent facial depth resolution (e.g., facial depth density) and can be used for post-processing. Then, method 500 can return to step 504.

[0218] refer to Figure 11 The diagram illustrates method (or process) 550. Method 550 determines the scaling ratio for each video frame. Method 550 generally includes steps (or states) 552, 554, decision step (or state) 556, 558, 560, 562, 564, 566, decision step (or state) 568, and step (or state) 570.

[0219] Step 552 can initiate method 550. In step 554, the scaling ratio module 314 can determine the number of depth pixels required for liveness detection in response to the signal EXP. Next, method 550 can move to decision step 556. In decision step 556, processor 102 can determine whether the user has updated the expected depth pixel tolerance. For example, the user can provide the input signal TOL to device 100 to adjust the tolerance value. Adjusting the tolerance value allows the user to calibrate the reduction between faster computation time (e.g., lower accuracy) and higher accuracy (e.g., slower computation time) within the expected depth pixel range. If the user has updated the tolerance value TOL, then method 550 can move to step 558. In step 558, processor 102 can update the expected depth pixel value. Next, method 550 can move to step 560. In decision step 556, if the user has not updated the expected depth pixel tolerance, then method 550 can move to step 560 (e.g., continuing to use the previous expected depth pixel value).

[0220] In step 560, the CNN module 190b can receive the next video frame from 302a-302n. Next, in step 562, the RGB extraction module 310 can extract RGB images 372a-372m from video frames 302a-302n, and then the face detection module 312 can determine the position of the face (e.g., offset values ​​LO1 / LO2 and TO1 / TO2) and the height and width of the face (e.g., dimensions FW1 / FW2 and FH1 / FH2). In step 564, the scaling module 314 can calculate the scaling ratio R of the current video frame (e.g., based on formula [1]). Then, in step 566, the scaling module 314 can provide the scaling ratio R to the reduction module 320. Next, method 550 can move to decision step 568.

[0221] In decision step 568, processor 102 can determine whether there are more video frames 302a-302n to process. The scaling ratio R can be recalculated for each video frame in video frames 302a-302n. For example, during the sequence of video frames 302a-302n, as the person captured in video frames 302a-302n moves in front of capture device 104 (e.g., forward, backward, left, right, up, down, etc.), various facial features (e.g., offset, height, width, etc.) can be updated. Processor 102 can recalculate the scaling ratio R for each video frame 302a-302n in real time. If there are more video frames 302a-302n to process, method 550 can return to decision step 556 (e.g., recalculate the scaling ratio R for the next video frame). If there are no more video frames 302a-302n to process, then method 550 can move to step 570. Step 570 can end method 550.

[0222] refer to Figure 12 The diagram illustrates method (or process) 600. Method 600 can perform post-processing on depth maps with consistent target dimensions. Method 600 generally includes steps (or states) 602, 604, a decision step (or state) 606, 608, 610, 612, 614, 616, and 618.

[0223] Step 602 can initiate method 600. In step 604, CNN module 190b can analyze the objects detected in RGB images 372a-372m. Next, method 600 can move to decision step 606. In decision step 606, CNN module 190b can determine whether a face has been detected in RGB images 372a-372m. For example, CNN module 190b can perform computer vision operations to detect objects and / or classify the detected objects to determine whether one or more faces exist in the video frame. If no face is detected, then method 600 can move to step 618. If a face has been detected, then method 600 can move to step 608.

[0224] In step 608, the face detection module 312 may record the face width (e.g., FW1), face height (e.g., FH1), and offset (e.g., TO1 and LO1) of the face 260 detected in the video frame. Next, in step 610, the cropping module 318 may crop the face from the full-resolution IR images 370a-370i based on the face data (e.g., FW1, FH1, TO1, LO1, etc.). In step 612, the reduction module 320 may reduce one of the corresponding cropped IR images 402a-402k based on a scaling ratio R (or reduce one of the full-resolution IR images 370a-370k if no cropping is performed). Next, in step 614, the depth map module 322 may generate depth maps DIMAGES from one of the cropped and reduced IR images 404a-404k. Due to the reduction and cropping, the depth maps DIMAGES may have a consistent target size (e.g., face depth density). In step 616, post-processing module 324 may perform liveness detection and / or other post-processing operations on the reduced depth maps DIMAGES. For example, post-processing module 324 may generate a signal LIVEOUT, which includes a result indicating whether the detected face is a real face (e.g., a real person) or a fake face (e.g., a facial photograph). Next, method 600 may proceed to step 618. Step 618 may end method 600.

[0225] refer to Figure 13 The diagram illustrates method (or process) 650. Method 650 can pre-calculate a reduced reference image. Method 650 generally includes steps (or states) 652, 654, 656, 658, 660, 662, decision steps (or states) 664, 666, 668, 670, and 672.

[0226] Step 652 can initiate method 650. In step 654, processor 102 and / or CNN module 190b can generate various sets of scaled reference images in response to full-resolution reference image 304 at a fixed scaling ratio. These various sets of scaled reference images can be generated before receiving real-time video frames 302a-302n. In the example, the fixed scaling ratio could be 1 / 2, 1 / 4, 1 / 8, 1 / 16, etc. Next, in step 656, the sets of scaled reference images can be stored in memory 150. Each set of reference images can be generated from the original full-resolution reference image 304 based on one of the fixed scaling ratios. Storing the sets of scaled reference images before receiving real-time video frames 302a-302n can provide faster computation time (e.g., compared to generating scaled reference images in real-time using a scaling ratio R calculated from the real-time input video frames). Storing the sets of scaled reference images in advance may incur the cost of memory storage requirements (e.g., using additional memory to store the sets of scaled reference images). Next, method 650 can be moved to step 658.

[0227] In step 658, processor 102 and / or CNN module 190b can analyze real-time input video frames 302a-302 (e.g., detect facial regions, extract RGB and IR images, etc.). Next, in step 660, scaling module 314 can calculate a real-time scaling ratio R in response to the input RGB image 372i, the size of the facial region, the expected number of depth pixels, and / or tolerance values. In step 662, processor 102 can compare the real-time scaling ratio value R with the next two fixed scaling ratio values. For example, if the fixed scaling ratio values ​​are 1, 1 / 2, 1 / 4, 1 / 8, etc., then the first comparison can be performed between the calculated scaling ratio R and 1 and 1 / 2, the second comparison can be performed between the calculated scaling ratio R and 1 / 2 and 1 / 4, the third comparison can be performed between the calculated scaling ratio R and 1 / 4 and 1 / 8, and so on. Next, method 650 can move to decision step 664.

[0228] In decision step 664, processor 102 may determine whether the real-time scaling ratio value R is within two fixed scaling ratio values. For example, processor 102 may determine whether ratio_p0 >= R > ratio_p1, where ratio_p0 and ratio_p1 are two fixed scaling ratios, and R is the real-time scaling ratio. If the real-time scaling ratio is not within the next two fixed scaling ratio values, then method 650 may return to step 662 and may perform the comparison again (e.g., using ratio_p1 and ratio_p2, then using ratio_p2 and ratio_p3, etc.). If the real-time scaling ratio is within the next two fixed scaling ratio values, then method 650 may move to step 666.

[0229] In step 666, processor 102 can modify the real-time scaling ratio value R to the higher of two selected fixed scaling ratio values ​​(e.g., if R is within ratio_p0 and ratio_p1, then R can then be modified to the same value as ratio_p0; if R is within ratio_p1 and ratio_p2, then R can then be modified to ratio_p1, and so on). In the example, if the real-time scaling ratio R is 1 / 1.5, and ratio_p0 is 1, and ratio_p1 is 1 / 2, then the real-time scaling ratio R can be modified to 1. In another example, if the real-time scaling ratio R is 1 / 3, and ratio_p1 is 1 / 2 and ratio_p2 is 1 / 4, then the real-time scaling ratio R can then be modified to 1 / 2. Next, in step 668, reduction module 320 can use the modified scaling ratio value to reduce the cropped IR image 402i. In step 670, depth map module 322 can generate depth maps DIMAGES from a set of reduced real-time image DLIMGs and reduced reference image DRIMGs at a selected reduction ratio. For example, reduction module 320 can reduce the cropped IR image 402i using a modified real-time scaling ratio value, and memory 150 can provide reduced reference images from a set of reduced reference images corresponding to the modified real-time scaling ratio value. Next, method 650 can proceed to step 672. Step 672 can end method 650.

[0230] One or more of the following can be used to achieve the effect of Figures 1-13The functions performed by the diagram are as follows: general-purpose processors, digital computers, microprocessors, microcontrollers, RISC (Reduced Instruction Set Computer) processors, CISC (Complex Instruction Set Computer) processors, SIMD (Single Instruction Multiple Data) processors, signal processors, central processing units (CPUs), arithmetic logic units (ALUs), video digital signal processors (VDSPs), and / or similar computing machines are programmed according to the teachings of this manual, as will be apparent to those skilled in the art. Appropriate software, firmware, codes, routines, instructions, opcodes, microcode, and / or program modules can be readily prepared by a skilled programmer based on the teachings of this disclosure, as will also be apparent to those skilled in the art. The software is typically implemented by one or more processors and executed from one or more media.

[0231] The present invention can also be implemented by preparing an ASIC (Application-Specific Integrated Circuit), a platform ASIC, an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), a CPLD (Complex Programmable Logic Device), a gate sea, an RFIC (Radio Frequency Integrated Circuit), an ASSP (Application-Specific Standard Product), one or more monolithic integrated circuits, one or more chips or dies arranged as flip-chip modules and / or multi-chip modules, or by interconnecting appropriate networks of conventional component circuits as described herein, modifications of which will be obvious to those skilled in the art.

[0232] Therefore, the present invention may also include a computer product, which may be a storage medium or medium and / or a transmission medium or medium, including instructions that can be used to program the machine to execute one or more processes or methods according to the present invention. The machine executes the instructions contained in the computer product and the operation of surrounding circuitry, and can convert input data into one or more files on the storage medium and / or one or more output signals representing physical objects or substances, such as audio and / or visual descriptions. The storage medium may include, but is not limited to, any type of disk, including floppy disks, hard disks, magnetic disks, optical disks, CD-ROMs, DVDs, and magneto-optical disks, as well as circuitry and / or any type of medium suitable for storing electronic instructions, such as ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Electrically Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), UVPROM (Ultraviolet Erasable Programmable ROM), flash memory, magnetic cards, optical cards, etc.

[0233] The elements of this invention can form part or all of one or more devices, units, components, systems, machines, and / or apparatuses. These devices may include, but are not limited to, servers, workstations, storage array controllers, storage systems, personal computers, laptop computers, notebook computers, handheld computers, cloud servers, personal digital assistants, portable electronic devices, battery-powered devices, set-top boxes, encoders, decoders, transcoders, compressors, decompressors, preprocessors, post-processors, transmitters, receivers, transceivers, cryptographic circuits, cellular phones, digital cameras, positioning and / or navigation systems, medical devices, head-up displays, wireless devices, audio recording, audio storage and / or audio playback devices, video recording, video storage and / or video playback devices, gaming platforms, peripheral devices, and / or multi-chip modules. Those skilled in the art will understand that the elements of this invention can be implemented in other types of devices to meet specific application standards.

[0234] The terms “may” and “generally” are used herein in conjunction with “is” and verbs to convey the intention that the description is exemplary and considered broad enough to encompass both the specific examples presented in this disclosure and the alternative examples that may be derived based on this disclosure. The terms “may” and “generally” as used herein should not be construed as necessarily implying the desirability or possibility of omitting the corresponding element.

[0235] The designations “a” through “n” for various components, modules, and / or circuits, when used herein, disclose a single component, module, and / or circuit or multiple such components, modules, and / or circuits, wherein the designation “n” is used to indicate any particular integer. Each distinct component, module, and / or circuit having an instance (or event) designated as “a” through “n” may indicate that the distinct component, module, and / or circuit may have a matching number of instances or a different number of instances. An instance designated as “a” may represent the first of multiple instances, and an instance “n” may refer to the last of multiple instances without implying a specific number of instances.

[0236] Although the invention has been specifically shown and described with reference to embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention.

Claims

1. An apparatus for generating a depth map, comprising: An interface configured to receive pixel data; A structured light projector configured to generate structured light patterns; as well as A processor configured to (i) process the pixel data arranged into a video frame including the structured light pattern, (ii) perform computer vision operations on the video frame to detect the size of a facial region in the video frame, (iii) determine a scaling ratio in response to the size of the facial region, (iv) extract the structured light pattern from the video frame, (v) generate a reduced structured light image, and (vi) generate a depth map in response to the reduced structured light image and a reduced reference image, wherein... (i) A reduction operation is performed in response to the scaling ratio and the structured light pattern to generate the reduced structured light image, and (ii) The scaling ratio is calculated such that the reduced structured light image can be generated with a sufficient number of depth pixels.

2. The apparatus according to claim 1, wherein, The reduced structured light image is generated with a sufficient number of depth pixels to prevent block matching failures when using the reduced structured light image.

3. The apparatus according to claim 1, wherein, The sufficient number of depth pixels provides the expected range of depth pixel counts for liveness detection operations.

4. The apparatus according to claim 1, wherein, The reduction operation is performed to reduce the number of calculations performed by the processor to generate the depth map using the reduced structured light image, compared to a second number of calculations performed to generate the depth map from the structured light pattern from the video frame.

5. The apparatus according to claim 1, wherein, The reduced reference image is generated in response to a reduction operation performed on the reference image.

6. The apparatus according to claim 1, wherein, (i) the reduced reference image is generated in response to a reduction operation performed on a plurality of reference images, (ii) the plurality of reference images are pre-calculated at a fixed reduction ratio, and (iii) the processor is further configured to (a) compare the scaling ratio with the fixed reduction ratio, and (b) modify the scaling ratio to match one of the fixed reduction ratios.

7. The apparatus according to claim 1, wherein, The processor is configured to perform adaptive generation of facial depth maps.

8. The apparatus according to claim 1, wherein, The scaling ratio is determined so that the depth map can be dynamically created within a resolution range corresponding to the size of the facial region.

9. The apparatus according to claim 1, wherein, The size of the facial region is a variable size in each video frame of the video frame, based on the distance between the person and the image sensor that captures the pixel data.

10. The apparatus according to claim 1, wherein, (i) The processor is configured to perform a face detection operation on the video frame to determine the size of the face region, and (ii) the face detection operation (a) determines whether the video frame captures at least one face, and (b) if at least one face is detected, records the offset and size of the face region.

11. The apparatus according to claim 1, wherein, (i) The scaling ratio is calculated in response to the sufficient number of depth pixels, the size of the facial region, and the tolerance parameter, and (ii) the size of the facial region includes facial width and facial height.

12. The apparatus according to claim 11, wherein, (i) The tolerance parameter is a configurable value set by the user, and (ii) the tolerance parameter allows the user to adjust the reduction operation between detection accuracy and calculation speed.

13. The apparatus according to claim 11, wherein, When the size of the facial region is large, the scaling ratio is a smaller value.

14. The apparatus according to claim 1, wherein, Compared to depth calculations that generate a depth map in response to the structured light pattern and a full-resolution reference image, the amount of time added to the calculations for performing the reduction operation is less than the time difference required to perform depth calculations in response to the reduced structured light image and the reduced reference image to generate the depth map.

15. The apparatus according to claim 1, wherein, The depth maps generated in response to the reduced structured light image and the reduced reference image have consistent facial depth resolution.

16. The apparatus according to claim 15, wherein, The consistent facial depth resolution of the depth map reduces the complexity of post-processing operations.

17. The apparatus of claim 16, wherein (i) the post-processing operation includes liveness detection, and (ii) the consistent facial depth resolution enables the liveness detection to be trained on facial depth having the consistent facial depth resolution rather than on facial depths with multiple resolutions.

18. The apparatus according to claim 1, wherein, The reduced structured light image has (i) a lower resolution compared to the structured light pattern and (ii) the same aspect ratio as the structured light pattern after the cropping operation.

19. The apparatus according to claim 1, wherein, (i) The facial region is detected by performing computer vision operations on a subset of the RGB values ​​of video frames captured when the structured light projector was not generating the structured light pattern, and (ii) The reduced structured light image is generated in response to a subset of the IR values ​​of video frames captured when the structured light projector was generating the structured light pattern.

20. The apparatus according to claim 19, wherein, The scaling ratio is determined in real time for each video frame in the RGB subset of the video frames so as to accommodate changes in the characteristics of the facial region as the person moves in front of the capture device that is generating the pixel data.

21. The apparatus of claim 1, wherein the apparatus is a camera system.

Images